mirror of
https://github.com/apache/cloudstack.git
synced 2025-12-22 05:24:54 +01:00
538 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Nicolas Vazquez
|
604137624d
|
FR01: Netris Integration (#1)
* Add Netris module and Add netris provider * Fix * Add Netris Provider to the zone creation wizard * add steps to zone wizard for adding netris controller and public traffic * cleanup * Add missing config key * Add routed mode offerings for Netris VPC (#3) * Add routed mode offerings for Netris VPC * update vpc offering name * generalize the offering creation method for network providers * log * remove debug log * fix failing build * Add dependency and Netris API client (#4) * Add dependency and first approach to Netris API client * Fix authentication and create Netris API client, in progress sites listing * Fix get sites * Support for listing VPCs (#5) * List tenants * Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack (#7) * Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack * Pass default auth scheme for now * Support adding netris provider to CloudStack and Netris VPC Creation (#6) * Support adding netris provider to CloudStack * revert marvin change * add license and perform session check when provider is added * add license and remove unused import * fix build failure - uunused imports * address comments * fix provider name * add Netris network element * add license * Add netris management APIs and netris service provider * add license * revert change * remove other network elements from Netris element * fix api name in doc generator * remove logs * move session alive check to CheckHealthCommand exec * Fix zone creation wizard to configure netris provider * Upgrade GSON version - from PR 8756 * Add additional parametes to the add Netris provider API * add netris as a host * add additional params to the resoponse and update UI * Rename site to site_name * Create Netris VPC (#8) * Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack (#7) * Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack * Pass default auth scheme for now * Drop for_nsx and for_tungten columns in favour of checking the provider on the ntwserviceprovider map table * Remove missing setForTungsten occurrence * Remove forNsx from VPC offerings * Create Netris VPC * Fix VPC offerings listing and remove unused dao * Create VPC fixes * Upgrade GSON version - from PR 8756 * Fix VPC creation response by using the latest SDK code * Fix unit test * Remove unused import * Fix NSX unit tests after refactoring * Add Netris key to the VLAN Details table (#10) * Add Netris key to the VLAN Details table * update for_<provider> column to be generic * Fix VPC and add IPAM allocation for the VPC CIDR (#9) * Fix VPC and add IPAM allocation for the VPC CIDR * Remove VPC logic * Use zoneId accountId and domainId on resources creation * Fix naming * Fix VR public nic issue * Fix Netris Public IP for VPC source NAT allocation * Add Netris VPC Subnets and vNets (#11) * Add Netris VPC Subnets and vNets * fix compilation errors * Add netris subnet * refactor naming convention to differentiate between VPC tiers and Isolated networks * revert marvin change * fix constructor - build failure * Add support to filter netris offerings, delete netris provider when zone is being deleted * Fix build * Fix VPC creation * Fix vnet creation * unnecesary log --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> --------- Co-authored-by: Pearl Dsilva <pearl1594@gmail.com> --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * Fix unit tests * Add support to delete VNets and Subnets (#13) * Add support to delete VNets and Subnets * Add support to delete vnet resources * Add support to delete vnet resources * extract code to method --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * Add missing suffix return (#14) * Set up Netris Public range on new zone addition (#15) * Set up Netris Public range on new zone addition * Add dependency to calculate subnet containing a start and end IP * Remove unused import * Move dependency to the netris module * Rename Netris IP range * Refactor logic * Revert "Refactor logic" This reverts commit 7ec36a81320444c37e7bb914dd895060b663411b. * Fix setup range after adding Netris Provider * Fix VXLAN range adding on zone creation * Pass VXLAN ID during creation of Netris vNets (#16) * add zone params to accepts management vnet * Release vxlan associated to the netris broadcast domain type * handle update network broadcast uri * Update Subnet purpose for Netris Public Traffic (#17) * Update Subnet purpose for Netris Public Traffic * search for existing subnet of common purpose type * Fix VR Public IP address (#20) * Fix VR Public IP address * Do not set the Public IP range on Netris side that is not part of the Netris IP Public Pool * Leave only systemvms tag for the first element * Fix NSX compatibility * Pass network gateway instead of network CIDR for Netris vNet creation (#21) * Run moodifyvxlan script if broadcast domain type is Netris (#18) * Add support to create Netris VPC / Network offerings (#22) * Add support to create Netris VPC / Network offerings * fix support services for netris provider type * Phase4 - Add support for Source NAT, Static NAT and Port Forwarding (#19) * Run moodifyvxlan script if broadcast domain type is Netris * Add Netris NAT offerings * Add support to add Source nat rules for Natted offering * fix api params while creating Netris source NAT rule * Add support to add and delete source nat rule on netris * Add support to create /32 NAT subnet * Add support to add and delete Static NAT rules in Netris (#23) * Add support to add and delete Static NAT rules in Netris * fix static nat creation on netris & removal of subnet on deletion of static nat rule * remove nat subnet after deltion of the static nat rule * add check to see if subnet already exists and add license header * Add port forwarding rules as DNAT rules in Netris (#24) * Add port forwarding rules as DNAT rules in Netris * Fixes * Allow removing DNAT rules * Fixes * Fix subnet search * Fix update SNAT only for SNAT rules * Address comments * Fix * Fix netris pom xml * Fix SNAT rule creation * Fix IP and port placements (#27) * Fix IP and port placements * fix dnat to IP for PF rules * change dnatport --------- Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com> * List only Netris Public IPs for NAT operations (#26) * List only Netris Public IPs for NAT operations * rename getter and change type * fix failing unit tests * list all IPs if forProvider is not passed * fix list public IPs for external providers with additional IP range * filter provider Ips in a zone with external provider setup * Prevent acquiring IP that is not from the external provider range * formating --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * Support to pass provider when creating public ip range and create IPAM on Netris (#28) * UI: support to pass provider when creating public ip range * prevent adding public ip range for a provider that isnt supported in zone * Create public range on Netris when created on CloudStack --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * Revert UI filtration for public IPs (#29) * Fix issue with pagination of public addresses listed after filtering for external providers * Revert UI filteration for public IPs for external network provider enabled zones * Fix unit tests (#30) * Add Netris Tag parameter to the Network provider and fix zone creation wizard (#33) * Add Netris Tag parameter to the Network provider * remove unused import * Fix public IP ranges creation on zone creation (#34) * use single quotes --------- Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com> * Fix SourceAddress for SNAt to VPC cidr (#35) * Fix VPC/network offering service list for external network providers in Routed mode (#32) * Fix network offering service list for external network providers in Routed mode * filter out unsupported services based on network mode * fix supported services list for vpc offering for external providers in Routed mode * Add support to add and delete and update static routes on Netris (#37) * Add support to add static routes in Netris * support to delete static routes on netris * add defensive check for nextHop * Add support to update static routes * add state * pass empty list for switched to avoid timeout * Netris: search static route by name and next hop if exists --------- Co-authored-by: Wei Zhou <weizhou@apache.org> * Netris FR1b: Support Remote Access VPN and Site-to-Site VPN in VPC VR (#41) * Static Routes: support nexthop * Update api/src/main/java/org/apache/cloudstack/api/command/user/vpc/CreateStaticRouteCmd.java Co-authored-by: Pearl Dsilva <pearl1594@gmail.com> * PR#10064 VR: apply iptables rules when add/remove static routes * PR#10065 UI: fix cannot open 'Edit tags' modal for static routes * PR#10066 Static Routes: fix check on wrong global configuration * PR#10067 VR: fix site-2-site VPN if split connections is enabled * PR#10081 server: do not allocate nic on public network for NSX VPC VR * PR#10082 UI: create VPC network offering with conserve mode * PR#10083 VR: allow outgoing traffic from RAS/VPN clients * PR#10086 server: fix typo removeaccessvpn in VirtualRouterElement * server: Add check on Public IP for remote access VPN * Revert "PR#10083 VR: allow outgoing traffic from RAS/VPN clients" This reverts commit 2f9b9f428947cac91de322fbdf4a980902a1c0a0. * VPC: fetch same used IP for domain router if VR is not Source NAT * VR: pass has_public_network to VR and configure RA/S2S VPN left peers * Revert "PR#10081 server: do not allocate nic on public network for NSX VPC VR" This reverts commit 809e269ed6b361d9df1fcef6537762c5612863e0. * VPC: fetch same used IP for domain router if VR is not Source NAT (v2) * VR: fix /etc/hosts and nameservers in dnsmasq.conf if VPC VR is not guest gateway prior to this PR ``` root@r-1167-VM:~# cat /etc/hosts 127.0.0.1 localhost 127.0.1.1 r-1167-VM ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters 172.21.1.33 dummy-vpc-vpn-001 172.21.1.1 r-1167-VM data-server root@r-1167-VM:~# cat /etc/dnsmasq.d/cloud.conf dhcp-hostsfile=/etc/dhcphosts.txt listen-address=127.0.0.1,172.21.1.234 dhcp-range=set:interface-eth1-0,172.21.1.234,static dhcp-option=tag:interface-eth1-0,15,cs2cloud.internal dhcp-option=tag:interface-eth1-0,6,172.21.1.1,10.0.32.1,8.8.8.8 dhcp-option=tag:interface-eth1-0,3,172.21.1.1 dhcp-option=eth1,26,1500 dhcp-option=tag:interface-eth1-0,1,255.255.255.0 ``` the lines should be ``` 172.21.1.234 r-1167-VM data-server dhcp-option=tag:interface-eth1-0,6,10.0.32.1,8.8.8.8 ``` * server: Enable static NAT for Domain router if it is not Source NAT * server: Enable static NAT for Domain router on UI * server: assign Public IP to VPC VR and enable static nat if VR is not Source NAT * server: configure dns1 if VR is not Source NAT * server: remove check on Firewall service when list network service providers * UI: remove dot from message.enabled.vpn * systemvm: add default route via first guest gateway if VR does not have public IP/interface * VR: add fw_dhcpserver for shared network * VR: pass has_public_network to VR and configure RA/S2S VPN left peers (v2) * UI: fix request error when create a VPC tier in a non-Netris/NSX env * systemvm: add default route via first guest gateway (v2) * VR: configure iptables rules for S2S vpn on first guest interface * VR: allow FORWARD to guest interfaces if VR is not Public * VR: configure remote access vpn on first guest interface if not public * VR: fix error 789 in RA VPN client when both RA and S2S are configured * server: Apply Static Route for RA/S2S VPN in VPC VR * VR: do not set mark for Public interface when VR is not really public * VPN: do not disable static nat if it is used by a RA/S2S VPN * server: skip check on network conserve mode if disable/enable RA VPN on Router IP * server: set forRouter to false when release a IP * VR: diable IP spoofing protection on default guest network * VR: fix iptables rules only when only S2S vpn is enabled * UI: show 'VPN Connections' section * VPC: new methods to configure/reconfigure Static NAT for VPC VR * API: set Type in ip address response to DomainRouter if it is used by VR * server: do not allow IP release if it is used by RA or S2S VPN gateway * VR: check if interface is added * VR: add default route only when ip is associated to first guest interface * VR: fix ipsec conf for l2tp and s2s vpn * server: save placeholder IP for VPC VR to fix the new VR IP when vpc tier is auto-shutdown * server: get non-placeholder NIC for VPC VR * VR: wait 15 seconds after starting password server * server: fix unable to configure static nat due to 'invalid virtual machine id' * UI: fix link of router in info card * VPC: apply static route for VPC VPN if needed (refactoring) * server: fix VR IP of first VPC tier is the VM gateway * server: update or remove all existing static routes when shutdown a network * server: update ipaddress after disabling static nat to fix vpc deletion issue * servr: disable remote access VPN as part of VPC dstroy * server: apply static routes when implement a vpc tier * server: apply static routes even if next hop is null * server: fix Cannot invoke "com.cloud.vm.NicProfile.getRequestedIPv4()" because "requested" is null * Netris: Update Vpn provider to VpcVirtualRouter * Netris: Add Vpn service to network offerings and networks * server: fix CIDR of VPN ip range * server: set isVrGuestGateway by SoureNat/Gateway service with Provider.VPCVirtualRouter * VR: password server takes 10-15 seconds to start if VR IP is not configured in /etc/hosts * Netris: add back routesPutBody.setStateStatus * engine/schema: remove SQL changes in schema-41910to42000.sql --------- Co-authored-by: Pearl Dsilva <pearl1594@gmail.com> * Add support for Gateway service for Netris VPC and network offerings (#39) * Add support for Gateway service for Netris VPC and network offerings * Restore UserData service * add gateway only to vpc service * Add support for gateway service for external network providers for networks in routed mode * add support for gateway svc * Revert "add support for gateway svc" This reverts commit 06645cd1c6d08a81ede5d1431497ea3f2efdc5dc. * Fix VPC offering creation * Fix VR public NIC after Gateway service is set to Netris --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * Netris VPN: Fix s2s vpn status update and isolated network implementation (#42) * server: fix NPE when deploy vm on isolated network * vpn: fix s2s vpn status is not updated Prior to this fix ``` java.lang.IllegalArgumentException: Class com.cloud.agent.api.CheckS2SVpnConnectionsAnswer declares multiple JSON fields named 'details'; conflict is caused by fields com.cloud.agent.api.CheckS2SVpnConnectionsAnswer#details and com.cloud.agent.api.Answer#details at com.cloud.agent.transport.ResponseTest.testCheckS2SVpnConnectionsAnswer(ResponseTest.java:42) ``` * test: fix test_01_vpn_usage as now it is only possible to create VPN on Source NAT if it uses VR * VR: fix unable to create remote access VPN on regular isolated network the error is ``` File "/opt/cloud/bin/configure.py", line 1242, in process self.remoteaccessvpn_iptables(self.dbag['public_interface'], public_ip, self.dbag[public_ip]) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ KeyError: 'public_interface' ``` * Release NAT IP subnet when VPC is removed or IP is released (#44) * Release NAT IP subnet when VPC is removed or IP is released * add license * Add support to add IPv6 Public IP range as IPAM Allocation / Subnet on Netris (#36) * Add support to add IPv6 Public IP range as IPAM Allocation / Subnet on Netris * Add ipam alloc and subnet for the ipv6 subnet associated to the vpc tier network * remove commented code * Phase5 - Support for ACLs in Netris (#31) * Add support for Netris ACLs * acl support * Make acl api call to netris to create the rule * refactor add acl rule to populate the right fields * support icmp type acl rule * acl rule creation - move netrisnetworkRule * Update ACL naming on Netris * Add support for Deletion of netris acls * Add support to delete and re-order ACL rules * support creation of default acl rules and replacing acl rules * fix NSXNetworkRule * Add global routing flag on subnet creation (#45) * Support change snat ip (#46) * Support updating VPC Source NAT IP * Optimize code * Update source NAT IP * Fix naming convention for NAT subnets to follow other resources (#47) * Fix naming convention for NAT subnets to follow other resources * Use vpc ID for nat subnets * Use new nat subnet name for deletion of static nat rule * fix naming convevntion for nat subnet * Keep Vpn service to default VPC offering with Natted mode only (#50) * Add Vpn service to default VPC offering with Routed mode * Revert change on VPC offering and fix VPN service only for Netris NAT mode * Validate if given CIDR belongs to a bigger allocation in Netris before creating the zone-level allocation (#48) * Validate if given CIDR belongs to a bigger allocation in Netris before creating * rename method * Phase5 - Support for LB - create, delete and Update operations (#49) * Add support for Netris ACLs * acl support * Make acl api call to netris to create the rule * refactor add acl rule to populate the right fields * support icmp type acl rule * acl rule creation - move netrisnetworkRule * Update ACL naming on Netris * Add support for Deletion of netris acls * Add support to delete and re-order ACL rules * support creation of default acl rules and replacing acl rules * fix NSXNetworkRule * Fix naming convention for NAT subnets to follow other resources * Use vpc ID for nat subnets * Phase5 - Support for LB - create, delete and Update operations * Use new nat subnet name for deletion of static nat rule * add support to add netris lb rule * support deletion of LB rule on Netris * add checks when editing unsupported fields of LB rule for Netris and hide columns on the UI * fix test failure * fix imports * add license * address comments * Enable Autoscaling on Netris for CPU and memory (#51) * Enable Autoscaling on Netris for CPU and memory * Fix monitor autoscale group and cleanup * Rename autoscaling group method * Integrate Autoscaling by allowing to update LB rules * Refactor according to the SDK changes * Fix the test failures noticed on #44 (#52) * Increase code coverage (#54) * Increase code coverage * More unit tests * Remove credentials and mock api client * NetrisResource tests * Fix unit test * Add support to add and remove ACL rules when CIDR list is passed when creating LB rules (#53) * Add support to add and remove ACL rules when CIDR list is passed when creating LB rules * add deny all rule * delete the deny rule as well * Fix build (#57) * Prevent Index Out of Bounds exception when naming IPAM subnets (#58) * Prevent Index Out of Bounds exception when naming IPAM subnets * fix linter * Delete netris IPv6 subnet (#59) * Netris VPN: add static route when update a non-existent static route (#60) * Fix VPC tier creation failure - prevent creating IPv6 IPAM allocation if it already exists (#61) * Update netris VPC and tier name (#56) * Update netris VPC and tier name * add support to update vpc tier name * add license * support editing names of dual stack VPCs * VR/server: configure default gateway and RA/S2S VPN on the IP/interface with minimum network_id (#43) * server: fix NPE when deploy vm on isolated network * vpn: fix s2s vpn status is not updated Prior to this fix ``` java.lang.IllegalArgumentException: Class com.cloud.agent.api.CheckS2SVpnConnectionsAnswer declares multiple JSON fields named 'details'; conflict is caused by fields com.cloud.agent.api.CheckS2SVpnConnectionsAnswer#details and com.cloud.agent.api.Answer#details at com.cloud.agent.transport.ResponseTest.testCheckS2SVpnConnectionsAnswer(ResponseTest.java:42) ``` * test: fix test_01_vpn_usage as now it is only possible to create VPN on Source NAT if it uses VR * VR: fix unable to create remote access VPN on regular isolated network the error is ``` File "/opt/cloud/bin/configure.py", line 1242, in process self.remoteaccessvpn_iptables(self.dbag['public_interface'], public_ip, self.dbag[public_ip]) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ KeyError: 'public_interface' ``` * VR/server: configure default gateway and RA/S2S VPN on the IP/interface with minimum network_id * Don't add deny rule if no CIDR list is passed (#62) * Hide the Stickiness Configure button for Netris Load Balancers (#72) * Update IPAM subnet purpose to nat before NAT operations if its different (#71) * Netris VPN: create vpc gateway with specified IP (#63) * Netris: fix UnsupportedOperationException when create VPC offering with NATTED mode (#75) fixes ``` 2025-03-21T10:42:55,039 ERROR [c.c.a.ApiServer] (qtp1513608173-21:[ctx-f9c7f002, ctx-bcfe846d]) (logid:e12e798f) unhandled exception executing api command: [Ljava.lang.String;@3a1416cd java.lang.UnsupportedOperationException at java.base/java.util.AbstractList.add(AbstractList.java:153) at java.base/java.util.AbstractList.add(AbstractList.java:111) at org.apache.cloudstack.api.command.admin.vpc.CreateVPCOfferingCmd.getServiceProviderMapForExternalProvider(CreateVPCOfferingCmd.java:248) ``` * [UI] Zone wizard creation improvements - rename hostname to url and remove port for Netris Provider (#77) * [UI] Zone wizard creation improvements - rename hostname to url and remove port for Netris Provider * Fix schema column for url instead of hostname * Fix Static NAT rules naming (#83) * Netris: create VPN gateway with specified public IP on UI (#82) * Netris vpn: apply static routes when start or delete a VPN connection (#85) * Netris VPN: apply static routes when start S2S VPN * Netris: list static routes and revoke the routes which are not needed * Netris: use route name (x.x.x.0/x) instead of prefix (x.x.x.0) and get clean cidr list * Netris VPN: fix NPE when list static routes * Update plugins/network-elements/netris/src/main/java/org/apache/cloudstack/service/NetrisApiClientImpl.java * Delete IPv6 allocation after tier removal on VPC with dual stack offering (#86) * Netris pass v6 gateway (#87) * pass v6 gateway to netris * pass v6 gateway to netris * refactor to address comments * remove imports * [VR] Fix IPv6 NIC IP on the VR (#89) * Add support to edit ACL rules (#74) * Add support to edit ACL rules * add support to update acl rules * remove test file * VR: advertise SLAAC prefix only if VR is gateway (#91) * Make reorder ACL items invoke Netris controller (#90) * VR: fix radvd misconfiguration for non-netris env (#92) * [VR] Fix object comparisson to string comparisson on python (#93) * Fix unit tests for ACL (#94) * Use the previously assigned vNet for Netris Network when it transitions to Implemented state after gc (#88) * Use the previously assigned vNet for Netris Network when it transitions to Implemented state after gc * Fix unit tests --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> --------- Co-authored-by: Pearl Dsilva <pearl1594@gmail.com> Co-authored-by: Wei Zhou <weizhou@apache.org> |
||
João Jandre
|
2fe3fcef7c |
Updating pom.xml version numbers for release 4.20.0.0
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com> |
||
Wei Zhou
|
679ce1a639
|
feature: Dynamic and Static Routing (#9470)
This PR contains 3 features - IPv4 Static Routing (Routed mode) #9346 Design document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=306153967 - AS Numbers Management #9410 Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/BGP+AS+Numbers+Management - Dynamic routing Design Document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=315492858 - Document: https://github.com/apache/cloudstack-documentation/pull/419 Rename nsx mode to routing mode by ``` git grep -l nsx_mode |xargs sed -i "s/nsx_mode/routing_mode/g" git grep -l nsxmode |xargs sed -i "s/nsxmode/routingmode/g" git grep -l nsxMode |xargs sed -i "s/nsxMode/routingMode/g" git grep -l NsxMode |xargs sed -i "s/NsxMode/RoutingMode/g" ``` - re-organize sql changes - fix NPE as rules do not have public ip - fix missing destination cidr in ingress rules - disable network usage for routed network - fix DB exception as network_id is -1 during network creation - apply ingress/egress routing rules - VR changes to configure nft rules for isolated network - VR: setup nft rule for control network - VR: flush all iptables rules - fix NPE which is because ingress rules do not have public ip associated - fix dest cidr is missing in nft tables - add ip4 routing and ip4 routes to list network and list vpc response - fix ingress rule is missing when vr is restarted - fix icmp types in nft rules - add tab to manage routing firewall rules - fix ingress rules are not applied when VR is restarted - add default rules in FORWARD chain - fix create vpc offerings - fix public ip is not assigned to vpc - fix network offering is not listed when create vpc tier - add is_routing to boot args of vpc vr - remove table ip4_firewall in vpc vr - release or remove subnet when remove a network - implemenent fw_vpcrouter_routing - fix wrong ip familty when flush ipv4 rules - fix acl rules are not applied due to wrong version (should be 6 which means ip6 rules are removed) - add default rules for vpc tiers so that tcp connections (e.g. ssh) work - append policy rules after default rules - remove /usr/local/cloud/systemvm/ in routers - throw an exception when allocate subnet with cidrsize - fix some TODOs - add new parameters to update API - return type Ipv4GuestSubnetNetworkMap when get or create subnet - fix firewall rules are broken - add domain_id and account_id to db - add domain/account/project to ipv4 subnet response - create ipv4 subnet for domain/account/project - check conflict when update ipv4 subnet - ui changes - add parent subnet to response - add list for ipv4 subnet - implement some methods - fix list subnets for guest networks by zoneid - UI changes - fix delete ipv4 subnet for network - fix ipv4 subnet is set to zone guest network cidr if cidrsize is specified - add zone info to response if parent subnet is null but network is not - fix gateway/cidr is not set when create network with cidrsize - fix order of nft rules in the VRs * Routed v24 - add classes in marvin base.py * Routed v25 - add test_01_subnet_zone - fix dedicate to domain/account failure - list subnets for network by keyword and subnet * Routed v26: implement subnet auto-allocation - add utils for split ip ranges into small subnets - add utils to get start/end ip of a cidr - implement subnet auto-generation - add global settings * Routed 27: add subnet for VPC - add db column for vpc_id - add db record for vpc - remove db record when delete a vpc - add checkConflicts methods - remove duplicated settings - check ipv4 cidr when create subnet * Routed v28: update smoke tests - update test_ipv4_routing.py - search subnets by networkid * Routed 29: fix vpc and add more tests - fix createnetwork in vpc - add vpc id/name to response - fix zone id/name are not displayed in some cases - add smoke test for vpc - add smoke tests for failed cases - add smoke test for connectivity checks - marvin: add "-q" to ssh command * Routed 31: ui and smoke tests - UI: add link to network in list view - add nftables rules check in VRs * Routed 32: add chain OUTPUT and more rules - fix the issue 80/443/8080 is not reachable from VR itself ``` 2024-06-27 10:21:52,121 INFO Executing: systemctl start cloud-password-server@172.31.1.1 2024-06-27 10:21:52,128 INFO Service cloud-password-server@172.31.1.1 start 2024-06-27 10:21:52,129 INFO Executing: ps aux 2024-06-27 10:24:02,175 ERROR Failed to update password server due to: <urlopen error [Errno 110] Connection timed out> ``` * Routed: fix dns search from VMs in Isolated networks * Routed: fix VPC dns issue due to gateway IP is missing in cloud.conf This is caused by NSX integration, and fixed by https://github.com/apache/cloudstack/pull/9102/ * Routed: rename routing_mode to network_mode * Routed: replace centos5.5 template in smoke test as dhclient does not work in the vms // this does not work refer to https://dominikrys.com/posts/disable-udp-checksum-validation/#ignoring-udp-checksums-with-nftables and https://forum.openwrt.org/t/udp-checksum-with-nftables/161522/11 the vm should have checksum offloading disabled * Routed: fix smoke test due to wrong cidrlist of egress rules and missing ingress rule from VR * PR 9346: fix lint error schema-41910to42000.sql * PR 9346: ui polish v1 * PR 9346: create VPC with cidrsize * Routed: fix test failures with test_network_ipv6 and test_vpc_ipv6 due to 'ssh -q' * Routed: fix /usr/local/cloud/systemvm/ are removed after SSVM/CPVM reboot * Routed: fix IP of additional nics of VPC VR is not gateway * PR 9346: fix cidrsize check when create VPC with cidrsize * Routed: fix test/integration/smoke/test_ipv4_routing.py:279:16: E713 test for membership should be 'not in' * PR9346: fix/Update api * PR 9346: set response object name * PR9346: UI refactor and small fixes * PR9346: change return type of getNetworkMode * PR9346: move IPv4 subnet to seperated tab * PR9346: revert IpRangesTabGuest.vue back to original * PR9346: fix remove ipv4 subnet on UI * PR9346: fix test_ipv4_routing.py * AS Number Range Management * Create AS Number Range for a Zone * Fix build * Add ListASNRange and fix create ASN range * Add List AS numbers * Add UI for AS Numbers * Fix UI and filter AS Numbers * Add AS Number on Isolated network creation and refactor UI and response * Release AS Number * Add network offering new columns * Add UI support to view and add AS number and configure network offering * Automatically assign AS Number if not specify AS number * update variable name * Fix routing mode check * UI: Only allow selecting AS number when routing mode is Dynamic and specifyAsNumber is true * UI: Only pass AS number when supported by the network offering * Release AS number on network deletion * Add deleteASNRange command (#81) * API: List ASNumbers by asnumber (#83) --------- Co-authored-by: Pearl Dsilva <pearl1594@gmail.com> * AS number management extensions * Support AS number on VPC tier creation based on the offering * Fix delete AS Range * Fix UI values * UI: Minor fix for releasing AS number * UI: Move management of AS Range to Zone details view * Fix specify_as_number column in network_offering table to set the default false * Add events for AS number operations * Allow users to list AS Numbers and fix network form for Normal users * Add AS number details to list networks response * Fix Allocated time format * Fix Allocated time format * support in details view too * Fix: Do not release AS number if acquired network requires AS number * Fix: Do not release AS number if acquired network requires AS number * Fix typo * Fix allocated release * Fix event type * UI: Add Routing mode and Specify AS to the network offering details * UI: Add Routing mode and Specify AS to the network offering details * Address comment * Fix release AS number of network deletion * Fix release AS number of network deletion * Fix * Restore release to its place based on the boolean * Rename boolean * API: Add networkId as listASNumber parameter * Add Network name to the search view filter for AS numbers * Present allocated time in human readable format - Pubilc IP / AS Numbers * Add account / domain filter for AS numbers * Add support for AS numbers on VPC offerings * Refactor AS number allocation to VPC and non VPC isolated networks * Checkstyle * Add support for AS numbers on VPC offerings * extend vpc offering view and vpcoffering response * merge https://github.com/shapeblue/cloudstack-playtika/pull/115 and change network_id of as_numbers to include vpc_id * Display AS number of VPC tiers as the AS number of the VPC * extend asnumber response and ui support * improve UI and as number response to view VPC details * List only dynamic offerings for vpc tiers with specify as numbers * Fix release AS number * Fix AS number displayed as 0 when no AS number assigned * Fix VPC offering creation without specify AS --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * Fix release AS number on VPC deletion * Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java * Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java * Fix missing column on asnumber table * Fix listASNumbers API to support vpcid and obtain AS number from vpc for tiers * Prevent listing 0 AS number for VPC * Fix create Isolated Network form * Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java * Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java * Dynamic: move routingmode/specifyasn after networkmode in AddNetworkOffering.vue on UI * Dynamic: fix ip4routing in network response * Dynamic/systemvm: add FRR to systemvm template * Dynamic: BGP peers (DB,VO,Dao) * Dynamic: BGP peers (VR/server) * Dynamic: v3 - remove BgpPeer class - fix vpc vr has bgp peers of only 1 tier - rename ip4_cidr to guest_ip4_cidr - rename ip6_cidr to guest_ip6_cidr - generate /etc/frr/frr.conf - apply BGP peers on Dynamic-Routed network even if there is no BGP peers * Dynamic v4: fix vpc vr - fix duplicated guest cidr in frr.conf in vpc vr todo - restart frr / reload frr (reload will cause bgp session to Policy state) - apis for bgp peers - assign/release bgp peer from/to network * Dynamic v5: add apis for bgp peers * Dynamic v6: fix bugs - set response object name - remove required as number when update - fix checks when update - allow regular users to list bgp peers * Dynamic v7: move apis to bgp sub-dir * Dynamic v8: add tab for manage BGP peers on UI * Dynamic v9: fix update bgp with same config * Dynamiv v10: add changeBgpPeersForNetworkCmd * Dynamic v11: create network with bgppeerids - create network with bgppeerids - add marvin classes - add smoke tests - remove uuid from bgp_peer_network_map - fix created/removed in bgp_peer_network_map - remove bgppeers when remove a network - UI: fix delete bgp peer * Dynamic v12: add test for vpc tiers * Dynamic v13: bug fixes - fix change BGP peers for network in Allocated state - fix listing network returns removed record - fix all vpc tiers have the same settings - remove BGP peers as part of network removal - remove FRR settings for vpc tiers without any BGP peers - UI: fix no error msg when change BGP peers * Dynamic v14: assign BGP Peers for VPC instead of VPC tiers - create vpc with bgppeerids - do not allow create/update vpc tier with bgppeerids - apply all bgp peers when create/delete a vpc tier - UI: change bgp peers for vpc - test: update tests on vpc * Dynamic: fix build errors after merging as number PR * Dynamic: fix TODOs * Dynamic: fix smoke test on VPC * Allow creation of networks by users with as numbers * Address review comments * Move BGPService to bgp package and inject it on BaseCmd * Revert changes for CKS and address more comments * Display left side menu option for AS number only for root admin * Dynamic: create/update BGP peer with details refer to https://docs.frrouting.org/en/latest/bgp.html * Dynamic: fix build error and remove access to ListBgpPeers cmd for regular users * Dynamic: assign all zone BGP peers to user networks * Dynamic: show BGP peer info of networks only for root admin * AS number: disable specifyasnumber for non-NSX offerings * Dynamic: pass bgppeer details to command and fix typo with ip6 addr * Dynamic: list BGP peers by isdedicated, and fix change bgppeers for network/vpc * Dynamic: add UI labels * Dynamic: add bgp peers to vpc response * Dynamic: list bgp peers by keyword, fix list by asnumber * Dynamic: fix list bgppeers by keyword and db schema * Dynamic: fix list bgppeers do not return dedicated peers * Dynamic: update UI when create network/vpc offering * Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com> * Update tools/marvin/setup.py * Dynamic: network mode must be same when update a network with new offering * Dynamic: add method networkModel.isAnyServiceSupportedInNetwork * Dynamic: rename APIs and classes * Dynamic: fix unit tests due to previous changes * Dynamic: validateNetworkCidrSize when auto-create subnet * Dynamic: check AS number overlap * Dynamic: add ActionEvent * Dynamic: small code optimization * Dynamic: fix ui bugs after api rename * Dynamic: add marvin and test for ASN ranges and AS numbers * Dynamic: add account setting use.system.bgp.peers also - change the default value of routed.ipv4.vpc.max.cidr.size and routed.ipv4.vpc.min.cidr.size - change the category of settings * static: fix ui error when delete zone ipv4 subnets * static: small UI polish * Dynamic: throw exception when as number is required but not passed * Dynamic: fix typo when create FRR directory which causes network deletion failures * Dynamic: connect to ALL (or ALL dedicated) BGP peers if no BGP peer mapping for the network/vpc * Dynamic: throw exception when as number is required for VPC but not passed * Dynamic: list bgp peers by useSystemBgpPeers * Dynamic: fix frr config in VPC VR when change bgp peers * Dynamic: create frr config even if there is no VPC tiers * Dynamic: list bgp peers by zoneid (required for account) and account * Dynamic: only apply FRR config for vpc tiers with dynamic routing * Dynamic: donot send commands to router if commands size is 0 * Dynamic: fix 'new IPv6 address is not valid' when update bgp peer without IPv6 * Dynamic: throw exception if fail to allocate AS number when create network/vpc with dynamic routing * Dynamic: enable ipv6 unicast and 'ip nht resolve-via-default' * Dynamic: delete network/vpc if fail to allocate AS number when create network/vpc with dynamic routing * test: add unit tests for ASN APIs * test: add unit tests for core module * test: add unit tests for API responses * test: add unit tests for BgpPeerTO * test: add minor changes * test: add tests for create/delete/update/list RoutingFirewallRuleCmd * Static: show ip4 routes for vpc tiers * test: fix smoke test failure caused by type change of as number * test: add test for Ipv4SubnetForZoneCmd * test: add test for Ipv4SubnetForGuestNetworkCmd and BgpPeerCmd * UI: do not show redundant router when network mode is ROUTED as RVR is not supported * UI: hide 'Conserve mode' when networkmode is ROUTED * test: add unit tests for ListASNumbersCmdTest * Static: remove allocated IPv4 subnet when delete a network or vpc * test: add unit tests for BgpPeersRules * Dynamic: set ipv4routing from network offering * server: list as numbers and ipv4 subnets by keyword * server: remove dedicated bgp peers and ipv4 subnets when delete an account or domain * server: fix dedicated ipv4 subnet is allocated to other accounts * UI: fix allocated time format * server: ignore project is projectid is -1 so bgppeers/ipv4subnets works in project view * UI: add project column to bgp peers and ipv4 subnets * server: fix list AS numbers by domain admin or normal user * server: fix network creation when ipv4 subnet is dedicated * UI: polish network.js * Dynamic: fix frr config for ipv6 routing * Static routing: support cks cluster * Static: get/create IPv4 subnet from dedicated subnets at first * Dynamic: add BGP peers tab * Static: remove redundant loops * api: add since to api and response * server: add unit tests --------- Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com> Co-authored-by: Pearl Dsilva <pearl1594@gmail.com> Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com> Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com> Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com> |
||
Wido den Hollander
|
c3f0d14d31
|
storage/object: Add support for Ceph RGW Object Store (#8389)
This feature adds support for Ceph's RADOS Gateway (RGW) support for the Object Store feature of CloudStack. The RGW of Ceph is Amazon S3 compliant and is therefor an easy and straigforward implementation of basic S3 features. Existing Ceph environments can have the RGW added as an additional feature to a cluster already providing RBD (Block Device) to a CloudStack environment. Introduce the BucketTO to pass to the drivers. This replaces just passing the bucket's name. Some upcoming drivers require more information then just the bucket name to perform their actions, for example they require the access and secret key which belong to the account of this bucket. This is leftover code from a long time ago and this validation test has nu influence on the end result on how a URL will be used afterwards. We should support hosts pointing to an IPv6(-only) address out of the box. For the code it does not matter if it's IPv4 or IPv6. This is the admin's choice. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com> Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com> |
||
Abhishek Kumar
|
0692a296ce
|
engine-orchestration: fix issue for empty product in vm metadata (#9610)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com> Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com> |
||
slavkap
|
12d9c26747
|
Added support for storpool_qos service (#8755) | ||
Vishesh
|
bc28665679
|
Add support for network data in Config Drive (#9329) | ||
|
Abhishek Kumar
|
1e12a80210
|
orchestration,hypervisor: allow custom manufacture, product for vm (#9163)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> |
||
John Bampton
|
c923e673cf
|
pre-commit: add XML files to the trailing-whitespace check (#9131)
|
||
|
Vishesh
|
0ec7c72875
|
Merge branch '4.19' | ||
Abhisar Sinha
|
063dc60114
|
Change storage pool scope from Cluster to Zone and vise versa (#8875)
* New feature: Change storage pool scope * Added checks for Ceph/RBD * Update op_host_capacity table on primary storage scope change * Storage pool scope change integration test * pull 8875 : Addressed review comments * Pull 8875: remove storage checks, AbstractPrimayStorageLifeCycleImpl class * Pull 8875: Fixed integration test failure * Pull 8875: Review comments * Pull 8875: review comments + broke changeStoragePoolScope into smaller functions * Added UT for changeStoragePoolScope * Rename AbstractPrimaryDataStoreLifeCycleImpl to BasePrimaryDataStoreLifeCycleImpl * Pull 8875: Dao review comments * Pull 8875: Rename changeStoragePoolScope.vue to ChangeStoragePoolScope.vue * Pull 8875: Created a new smokes test file + A single warning msg in ui * Pull 8875: Added cleanup in test_primary_storage_scope.py * Pull 8875: Type in en.json * Pull 8875: cleanup array in test_primary_storage_scope.py * Pull:8875 Removing extra whitespace at eof of StorageManagerImplTest * Pull 8875: Added UT for PrimaryDataStoreHelper and BasePrimaryDataStoreLifeCycleImpl * Pull 8875: Added license header * Pull 8875: Fixed sql query for vmstates * Pull 8875: Changed icon plus info on disabled mode in apidoc * Pull 8875: Change scope should not work for local storage * Pull 8875: Change scope completion event * Pull 8875: Added api findAffectedVmsForStorageScopeChange * Pull 8875: Added UT for findAffectedVmsForStorageScopeChange and removed listByPoolIdVMStatesNotInCluster * Pull 8875: Review comments + Vm name in response * Pull 8875: listByVmsNotInClusterUsingPool was returning duplicate VM entries because of multiple volumes in the VM satisfying the criteria * Pull 8875: fixed listAffectedVmsForStorageScopeChange UT * listAffectedVmsForStorageScopeChange should work if the pool is not disabled * Fix listAffectedVmsForStorageScopeChangeTest UT * Pull 8875: add volume.removed not null check in VmsNotInClusterUsingPool query * Pull 8875: minor refactoring in changeStoragePoolScopeToCluster * Update server/src/main/java/com/cloud/storage/StorageManagerImpl.java * fix eof * changeStoragePoolScopeToZone should connect pool to all Up hosts Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com> |
||
Suresh Kumar Anaparti
|
2ca1b474bd
|
PowerFlex/ScaleIO SDC client connection improvements (#9268)
* Mitigation for non-scalable Powerflex/ScaleIO clients - Added ScaleIOSDCManager to manage SDC connections, checks clients limit, prepare and unprepare SDC on the hosts. - Added commands for prepare and unprepare storage clients to prepare/start and stop SDC service respectively on the hosts. - Introduced config 'storage.pool.connected.clients.limit' at storage level for client limits, currently support for Powerflex only. * tests issue fixed * refactor / improvements * lock with powerflex systemid while checking connections limit * updated powerflex systemid lock to hold till sdc preparation * Added custom stats support for storage pool, through listStoragePools API * code improvements, and unit tests * unit tests fixes * Update config 'storage.pool.connected.clients.limit' to dynamic, and some improvements * Stop SDC on host after migration if no volumes mapped to host * Wait for SDC to connect after scini service start, and some log improvements * Do not throw exception (log it) when SDC is not connected while revoking access for the powerflex volume * some log improvements |
||
|
Wei Zhou
|
d81ffd2d86
|
Merge remote-tracking branch 'apache/4.19' | ||
|
Abhishek Kumar
|
3e6900ac1a
|
api,server: purge expunged resources (#8999)
This PR introduces the functionality of purging removed DB entries for CloudStack entities (currently only for VirtualMachine). There would be three mechanisms for purging removed resources:
Background task - CloudStack will run a background task which runs at a defined interval. Other parameters for this task can be controlled with new global settings.
API - New admin-only API purgeExpungedResources. It will allow passing the following parameters - resourcetype, batchsize, startdate, enddate. Currently, API is not supported in the UI.
Config for service offering. Service offerings can be created with purgeresources parameter which would allow purging resources immediately on expunge.
Following new global settings have been added:
expunged.resources.purge.enabled: Default: false. Whether to run a background task to purge the expunged resources
expunged.resources.purge.resources: Default: (empty). A comma-separated list of resource types that will be considered by the background task to purge the expunged resources. Currently only VirtualMachine is supported. An empty "value will result in considering all resource types for purging
expunged.resources.purge.interval: Default: 86400. Interval (in seconds) for the background task to purge the expunged resources
expunged.resources.purge.delay: Default: 300. Initial delay (in seconds) to start the background task to purge the expunged resources task.
expunged.resources.purge.batch.size: Default: 50. Batch size to be used during expunged resources purging.
expunged.resources.purge.start.time: Default: (empty). Start time to be used by the background task to purge the expunged resources. Use format yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.
expunged.resources.purge.keep.past.days: Default: 30. The number of days in the past from the execution time of the background task to purge the expunged resources for which the expunged resources must not be purged. To enable purging expunged resource till the execution of the background task, set the value to zero.
expunged.resource.purge.job.delay: Default: 180. Delay (in seconds) to execute the purging of an expunged resource initiated by the configuration in the offering. Minimum value should be 180 seconds and if a lower value is set then the minimum value will be used.
Documentation PR: apache/cloudstack-documentation#397
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
|
||
Harikrishna
|
2315a73a20
|
User friendly name of Downloaded Templates Volumes and ISOs (#9252) | ||
Daan Hoogland
|
373f017002 | Merge branch '4.19' | ||
|
Harikrishna
|
bb0c1f93af
|
Add volume encryption checks during the disk offering change (#9209) | ||
|
Vishesh
|
21af134087
|
Fix exceeding of resource limits with powerflex (#9008)
* Fix exceeding of resource limits with powerflex * Add e2e tests * Update server/src/main/java/com/cloud/vm/UserVmManagerImpl.java Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com> * fixup --------- Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com> |
||
|
Daan Hoogland
|
0d8f7d4003 |
Merge release branch 4.19 to main
* 4.19: linstor: disconnect-disk also search for resource name in Linstor (#9035) ui: add support to change Account role for admins (#9012) Use parameter dcId as wrapper to prevent NPE (#8986) |
||
|
dahn
|
e520525fe7
|
Use parameter dcId as wrapper to prevent NPE (#8986) | ||
|
Daan Hoogland
|
e61f3bae4d | Merge branch '4.19' | ||
|
Vishesh
|
80a8b80a9d
|
Update volume's passphrase to null if diskOffering doesn't support encryption (#8904) | ||
|
Vishesh
|
cfdb33a052
|
Fixup resource limit checks (#8935) | ||
|
Daan Hoogland
|
a358c9a410 |
Merge release branch 4.19 to main
* 4.19: New feature: Import/Unamange DATA volume from storage pool (#8808) |
||
|
Wei Zhou
|
0b857def68
|
New feature: Import/Unamange DATA volume from storage pool (#8808) | ||
|
Wei Zhou
|
45daa1ce59
|
Merge remote-tracking branch 'apache/4.19' | ||
|
Vishesh
|
b998e7dbb6
|
Allow overriding root disk offering & size, and expunge old root disk while restoring a VM (#8800)
* Allow overriding root diskoffering id & size while restoring VM * UI changes * Allow expunging of old disk while restoring a VM * Resolve comments * Address comments * Duplicate volume's details while duplicating volume * Allow setting IOPS for the new volume * minor cleanup * fixup * Add checks for template size * Replace strings for IOPS with constants * Fix saveVolumeDetails method * Fixup * Fixup UI styling |
||
|
Vishesh
|
19f79b1d94
|
Merge branch '4.19' | ||
|
Vishesh
|
730cc5d5b8
|
Change iops on offering change (#8872)
* Change IOPS on disk offering change * Remove iops & bandwidth limits before copying template * minor refactor * Handle diskOfferingDetails * Fixup |
||
|
Abhishek Kumar
|
02305fbc5f | Merge remote-tracking branch 'apache/4.19' | ||
|
Wei Zhou
|
939d0b9011 |
engine-storage: control download redirection
Add a global setting to control whether redirection is allowed while downloading templates and volumes core: some changes on SimpleHttpMultiFileDownloader similar as HttpTemplateDownloader Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> (cherry picked from commit b1642bc3bf58ccde9f56f632b5a9fe46a3eb5356) Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com> |
||
|
Vishesh
|
33dc7465c2
|
Merge remote-tracking branch 'origin/4.19' | ||
|
Abhishek Kumar
|
ffd59720dd
|
storage,plugins: delegate allow zone-wide volume migration check and access grant check to storage drivers (#8762)
* storage,plugins: delegate allow zone-wide volume migration check and access grant to storage drivers Following checks have been delegated to storage drivers, - For volumes on zone-wide storage, whether they need storage migration when VM is migrated - Whther volume required grant access Apply fixes in resolving PrimaryDataStore * add tests Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> * unused import Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> * Update engine/orchestration/src/test/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestratorTest.java --------- Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> |
||
Pearl Dsilva
|
6dc3d06037
|
NSX integration (#7919)
* NSX integration - skeletal code * Fix module not loading on startup * add upgrade path and daos \n add nsx controller command * add support for adding and listing nsx provider to a zone * add license * add default VPC offering and update upgrade path * add global setting to enable nsx plugin * add delete nsx controller operation * add nsxresource * add NSX resource , api client, create tier1 gw * update db * update response and add license * Add support to create and delete nsx tier-1 gateway * add license * cleanup and add skeletal code for network creation * add create/delete segment and UI integration * add license * address code smells - part 1 * fix test / build failure * NSX integration - skeletal code * Fix module not loading on startup * add upgrade path and daos \n add nsx controller command * add support for adding and listing nsx provider to a zone * add license * add default VPC offering and update upgrade path * add global setting to enable nsx plugin * add delete nsx controller operation * add nsxresource * add NSX resource , api client, create tier1 gw * update db * update response and add license * Add support to create and delete nsx tier-1 gateway * add license * cleanup and add skeletal code for network creation * add create/delete segment and UI integration * add license * address code smells - part 1 * fix test / build failure * add ui changes + update nsx_provider table transport zones + use NSX broadcast domain for add nics to router * ui: fix password field, and backend changes * add route advertisement * update offering * update offering * add sleep before deletion of vpc / tier g/w for ports to be removed * move creation of segments to design phase * change provider to VPC router for Dhcp & dns service in an nsx offering * Add public nic for NSX * reserve first IP (after g/w) of subnet for router nic - NSX * revert reserving 1st IP in vpc segments * [NSX] Create a DHCP relay and add it to a VPC tier segment (#107) * Create DHCP relay command and execute request * In progress integrate with networking * Create DHCP relay config on the network VR allocation * Revert domain router dao changes * Create DHCP relay con VR nic plug to NSX network * Link DHCP relay config to segment after creation * [NSX] Cleanup DHCP Relay config on segment deletion (#108) * Cleanup DHCP Relay config on segment deletion * update segment & relay name generators and call delete dhcprelay after deletion of segment * address comment * [NSX] Fix DHCP relay config deletion was missing zone name (#8068) * [NSX] Refactor API wrapper operations (#8059) * [NSX] Refactor API wrapper operations * Big refactor * Address review comment * change network cidr to cidr to prevent NPE * add domain and zone names to the various networks - vpc & tier --------- Co-authored-by: Pearl Dsilva <pearl1594@gmail.com> * Nsx unit tests (#8090) * Add tests * add test for NsxGuestNetworkGuru * add unit tests for NsxResource * add unti tests for NsxElement * cleanup * [NSX] Refactor API wrapper operations * update tests * update tests - add nsxProviderServiceImpl test * add unit test - NsxServiceImpl * add license * Big refactor * Address review comment * change network cidr to cidr to prevent NPE * add domain and zone names to the various networks - vpc & tier * fix tests --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * modify NSX resource naming convention (#8095) * modify NSX resource naming convention * remove unused imports * add a setup phase between desgin and implementation of a network for intermediary steps * add method to all classes * NSX: Refactor Network & VPC offering (#8110) * [NSX] Refactor API wrapper operations * Network offering changes for NSX * fix services and provider combination * address comments: rename param * update nsx_mode parameter --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * fix test * [NSX] Allow NSX isolated networks (#8132) * Add network offerings for NSX on isolated networks * Fix offerings creation * In progress NSX isolated network * Fixes * Fix NIC allocation to router * NSX: Add Step for Adding Public traffic network for NSX During zone creation (#8126) * NSX: Add Step for Adding Public traffic network for NSX * address comments and cleanup * address comment * remove indent * NSX: Create and Delete static NAT & Port forward rules (#8131) * NSX: Create and delete NSX Static Nat rules * fix issues with static nat * add static nat * Support to add and delete Port forward rules * add license * fix adding multiple pf rules * cleanup * fix lint check * fix smoke tests * fix smoke tests * Nsx add lb rule (#8161) * NSX: Create and delete NSX Static Nat rules * fix issues with static nat * add static nat * Support to add and delete Port forward rules * add license * fix adding multiple pf rules * cleanup * NSX: Add support to create and delete Load balancer rules * fix deletion of lb rules * add header file and update protocol detail * build failure fix * [NSX] Add SNAT support (#8100) * In progress add source NAT * Fix after merge * Fix tests * Fix NPE on isolated network deletion * Reserve source NAT IP when its not passed for NSX VPC * Create source NAT rule on VR NIC allocation * Fix update VPC and remove VPC to update and remove SNAT rule * Fix packaging * Address review comment * Fix build * fix build - unused import * Add defensive checks * Add missing design to NSX public guru --------- Co-authored-by: Pearl Dsilva <pearl1594@gmail.com> * NSX: Fix VR public NIC allocation (#8166) * NSX: fix LB member addition and deletion and add defensive checks (#8167) * Fix public NIC NPE on broadcast URI * NSX: Router Public nic to get IP from systemVM Ip range (#8172) * NSX: Router Public nic to get IP from systemVM Ip range * Fix VR IP address and setSourceNatIp command * NSX: hide systemVM reserved IP range SourceNAT * fix test --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * fix test failure * test failure fix * [NSX] Fix update source NAT IP (#8176) * [NSX] Fix update source NAT IP * Fix startup * Fix API result * NSX - add LB route Advertizement (#8192) * [NSX] Add ACL types support (#8224) * NSX: Create segment group on segment creation * Add unit tests * Remove group for segment before removing segment * Create Distributed Firewall rules * Remove distributed firewall policy on segment deletion * Fix policy rule ID and add more unit tests * Fix DROP action rules and transform tests * Add new ACL rules * Fixes * associate security policies with groups and not to DFW and add deletion of rules * Fix name convention --------- Co-authored-by: Pearl Dsilva <pearl1594@gmail.com> * NSX: Fix creation of VPCs (#8320) * Fix ACL rules creation (#8323) * [NSX] Fix database views (#8325) * NSX: Add CKS Support & Firewall rules for Isolated Networks (#8189) * NSX: Add ALL LB IP to the list of route advertisements in tier1 * NSX: Support Source NAT on NSX Isolated networks * NSX: Cks Support * NSX: Create segment group on segment creation * Add unit tests * Remove group for segment before removing segment * Create Distributed Firewall rules * Remove distributed firewall policy on segment deletion * Fix policy rule ID and add more unit tests * Add support for routed NSX Isolated networks \n and non RFC 1918 compliant IPs * Add support for routed NSX Isolated networks \n and non RFC 1918 compliant IPs * Add Firewall rules * build failure - fix unit test * fix npes * Add support to delete firewall rules * update nsx cks offering * add license * update order of ports in PF & FW rules * fix filter for getting transport zones * CKS support changed - MTU updated, etc * add LB for CKS on VPC * address comments * adapt upstream cks logic for vpc * rever mtu hack * update UI changes as per upstream fix * change display test for CKS n/w offerings for isolated and VPC tiers * add extra line for linter * address comment * revert list change --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * fix ui build failure * [NSX] Address SonarCloud Bugs (#8341) * [NSX] Address SonarCloud Bugs * Fix NSX API connection issues * NSX: Add unit tests to increase coverage (#8355) * NSX: Add unit tests * cleanup unused imports * add more unit tests * add tests for publicnsxnetworkguru * add license * fix build failures * address sonar comment * fix security hotspots * NSX: Add more unit tests (#8381) * NSX : Unit tests * remove unused imports * remove unused import causing build failure * fix build failures due to unused imports * fix build failure * fix test assertion * remove unused imports * remove unused import * Nsx UI zone bug (#8398) * NSX: Attempt to fix NSX Zone creation bug for public networks * fix zone wizard public traffic issue * add proper filtering of offerings based on VPC nsx mode * clean up console logs * NSX: Fix code smells and reported bugs (#8409) * NSX: Fix code smells and reported bugs * fox override issue * remove unused imports * fix test * refactor code to reduce complexity * add lisence * cleanup * fix build failure * fix build failure * address comments * test - add config to ignore certain files from test coverage * test exclusion of classes from test cov * rever pom changes * [NSX] Add more unit tests (#8431) * [NSX] Add more unit tests * More tests * Fix build errors * NSX: Prevent creation of L2 and Shared networks for NSX (#8463) * NSX: Prevent creation of L2 and Shared networks for NSX * add checks to backend to prevent creation of l2 and shared networks in nsx zones and filter only nsx offerings when creating isolated networks * cleanup * NSX: Fix code smells (#8436) * NSX: Fix code smells * Add changes to service creation logic * CKS: Add action to during firewall rule creation (#8498) * NSX,UI: Deduplicate network list when creating kubernetes clusters (#8513) * NSX: Make LB service selectable in network offering (#8512) * NSX: Make LB service selectable in network offering * fix label * address comments * address comments * NSX: Add appropriate error message when icmp type is set to -1 for NSX (#8504) * NSX: Add appropriate error message when icmp type is set to -1 for NSX * address comments * update text * fix test * fix test - build failure * fix test - build failure * NSX: Cleanup NSX resources during k8s cluster cleanup (#8528) * fix test failure * NSX: Improve segment deletion process (#8538) * NSX: Add passive monitor for NSX LB to test whether a server is available (#8533) * NSX: Add passive monitor for NSX LB to test whether a server is available * Add active monitors too * fix build failure * NSX: Add check for ICMP code / type for NSX zones (#8542) * NSX: Fix Routed Mode for Isolated and VPC networks (#8534) * NSX: Fix Routed Mode for Isolated and VPC networks * NSX: Fix Routed mode - add checks for ports added for FW rules * clean up code * fix build failure * NSX: Add retry logic with sleep to delete segments (#8554) * NSX: Add retry logic with sleep to delete segments * add logs * NSX: Fix custom ACL check (#2) * NSX: Fix custom ACL check * NSX: Fix custom ACL check * Nsx vpc routed mode (#5) * NSX: Fix VPC routed mode * NSX: VPC route mode * remove unnecessary changes * Nsx: Support internal LB (#4) * NSX: Support internal LB service in NSX * add lb removal logic * Fix UI issue hiding internal LB tab * Refactor method name --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * NSX: Improve NSX resource cleanup process (#3) * Fix unit test * NSX: Add SourceNAT service to the default Routed offering for VPC (#13) * Fix VPC restart with cleanup (#12) * NSX: Fix ACL rule removal on replacement and fix rule order (#11) * NSX: fix smoke test failure for ACLs (#9) * Fix unit tests * Fix NSX plugin pom XML * NSX: Add support to re-order ACL rules (NSX FW rules) (#14) * [WIP] NSX: Add support to re-order ACL rules (NSX FW rules) * fix reordering of acl rules on all networks that it is associated to * clean up and attempt test fix * Fix tests * Remove unused import * tweak reorder logic --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> * Fix zone creation issue for internal load balancer * Fix * Fix unit test * fix logger * fix logger * fix logger * NSX: Fix VPC form to ignore source NAT IP when creating VPCs and fix label * Move SQL changes to the newest schema file * NSX: Last Fixes * Fix build --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> |
||
|
Abhishek Kumar
|
b29ec2bf12 | Merge remote-tracking branch 'apache/4.19' | ||
|
Harikrishna
|
c462be1412
|
New API "checkVolume" to check and repair any leaks or issues reported by qemu-img check (#8577)
* Introduced a new API checkVolumeAndRepair that allows users or admins to check and repair if any leaks observed. Currently this is supported only for KVM * some fixes * Added unit tests * addressed review comments * add repair volume while granting access * Changed repair parameter to accept both leaks/all * Introduced new global setting volume.check.and.repair.before.use to do volume check and repair before VM start or volume attach operations * Added volume check and repair changes only during VM start and volume attach operations * Refactored the names to look similar across the code * Some code fixes * remove unused code * Renamed repair values * Fixed unit tests * changed version * Address review comments * Code refactored * used volume name in logs * Changed the API to Async and the setting scope to storage pool * Fixed exit value handling with check volume command * Fixed storage scope to the setting * Fix volume format issues * Refactored the log messages * Fix formatting |
||
|
Daan Hoogland
|
3baa45bc2a | forward Merge branch '4.19' into main | ||
|
Daan Hoogland
|
f4987bf8ee |
Merge release branch 4.18 to 4.19
* 4.18: Storage plugin support to check if volume on datastore requires access for migration (#8655) CKS: fix /opt/bin/deploy-cloudstack-secret in CKS control nodes (#8697) |
||
Suresh Kumar Anaparti
|
f731fe882c
|
Storage plugin support to check if volume on datastore requires access for migration (#8655)
* Check if volume on datastore requires access for migration, and grant/revoke volume access if requires * Updated default implementation for requiresAccessForMigration method in PrimaryDataStoreDriver |
||
|
Wei Zhou
|
87284f03f0
|
Upgrade to JRE17 and Upgrade System VMs/VRs to Python3 and Debian 12 (#8497)
* Update to 4.20.0
* Update to python3
* Upgrade to JRE 17
* Upgrade to Debian 12.4.0
* VR: upgrade to python3
for f in `find systemvm/ -name *.py`;do
if grep "print " $f >/dev/null;then
2to3-2.7 -w $f
else
2to3-2.7 -p -w $f
fi
done
* java: Use JRE17 in cloudstack packages and systemvmtemplate
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Add --add-opens to JAVA_OPTS in systemd config
* Add --add-opens to JAVA_OPTS in systemd config for usage
* python3: fix "TypeError: a bytes-like object is required, not 'str'"
* python3: fix "ValueError: must have exactly one of create/read/write/append mode"
* Add --add-exports=java.base/sun.security.x509=ALL-UNNAMED for management server
* Use pip3 instead of pip for centos8
* python3: fix "TypeError: write() argument must be str, not bytes"
```
root@r-1037-VM:~# /opt/cloud/bin/passwd_server_ip.py 10.1.1.1
Traceback (most recent call last):
File "/opt/cloud/bin/passwd_server_ip.py", line 201, in <module>
serve()
File "/opt/cloud/bin/passwd_server_ip.py", line 187, in serve
initToken()
File "/opt/cloud/bin/passwd_server_ip.py", line 60, in initToken
f.write(secureToken)
TypeError: write() argument must be str, not bytes
root@r-1037-VM:~#
```
* Python3: fix "name 'file' is not defined"
```
root@r-1037-VM:~# /opt/cloud/bin/passwd_server_ip.py 10.1.1.1
Traceback (most recent call last):
File "/opt/cloud/bin/passwd_server_ip.py", line 201, in <module>
serve()
File "/opt/cloud/bin/passwd_server_ip.py", line 188, in serve
loadPasswordFile()
File "/opt/cloud/bin/passwd_server_ip.py", line 67, in loadPasswordFile
with file(getPasswordFile()) as f:
NameError: name 'file' is not defined
```
* python3: fix "TypeError: write() argument must be str, not bytes" (two more files)
* Upgrade jaxb version
* python3: fix more "TypeError: a bytes-like object is required, not str"
* python3: fix "Failed to update password server"
Failed to update password server due to: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.
* python3: fix "bad duration value: ikelifetime=24.0h"
Jan 15 13:57:20 systemvm ipsec[3080]: # bad duration value: ikelifetime=24.0h
* python3: fix password server "invalid save_password token"
* test: incease retries in test_vpc_vpn.py
* python3: fix passwd_server_ip.py
see error below
```
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: ----------------------------------------
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: Exception occurred during processing of request from ('10.1.1.129', 32782)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: Traceback (most recent call last):
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: File "/usr/lib/python3.9/socketserver.py", line 650, in process_request_thread
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: self.finish_request(request, client_address)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: File "/usr/lib/python3.9/socketserver.py", line 360, in finish_request
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: self.RequestHandlerClass(request, client_address, self)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: File "/usr/lib/python3.9/socketserver.py", line 720, in __init__
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: self.handle()
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: File "/usr/lib/python3.9/http/server.py", line 427, in handle
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: self.handle_one_request()
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: File "/usr/lib/python3.9/http/server.py", line 415, in handle_one_request
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: method()
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: File "/opt/cloud/bin/passwd_server_ip.py", line 120, in do_GET
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: self.wfile.write(password)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: File "/usr/lib/python3.9/socketserver.py", line 799, in write
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: self._sock.sendall(b)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: TypeError: a bytes-like object is required, not 'str'
```
* python3: fix self.cl.get_router_password in Redundant VRs
```
File "/opt/cloud/bin/cs/CsDatabag.py", line 154, in get_router_password
md5.update(passwd)
TypeError: Unicode-objects must be encoded before hashing"]
```
* scripts: mark multipath scripts as executable
* systemvm template: remove hyperv packages and do not export
* VR: update default RAM size of System VMs/VRs to 512MiB
Before
```
mysql> select id,name,cpu,speed,ram_size,unique_name,system_use from service_offering where name like "System%";
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
| id | name | cpu | speed | ram_size | unique_name | system_use |
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
| 3 | System Offering For Software Router | 1 | 500 | 256 | Cloud.Com-SoftwareRouter | 1 |
| 4 | System Offering For Software Router - Local Storage | 1 | 500 | 256 | Cloud.Com-SoftwareRouter-Local | 1 |
| 5 | System Offering For Internal LB VM | 1 | 256 | 256 | Cloud.Com-InternalLBVm | 1 |
| 6 | System Offering For Internal LB VM - Local Storage | 1 | 256 | 256 | Cloud.Com-InternalLBVm-Local | 1 |
| 7 | System Offering For Console Proxy | 1 | 500 | 1024 | Cloud.com-ConsoleProxy | 1 |
| 8 | System Offering For Console Proxy - Local Storage | 1 | 500 | 1024 | Cloud.com-ConsoleProxy-Local | 1 |
| 9 | System Offering For Secondary Storage VM | 1 | 500 | 512 | Cloud.com-SecondaryStorage | 1 |
| 10 | System Offering For Secondary Storage VM - Local Storage | 1 | 500 | 512 | Cloud.com-SecondaryStorage-Local | 1 |
| 11 | System Offering For Elastic LB VM | 1 | 128 | 128 | Cloud.Com-ElasticLBVm | 1 |
| 12 | System Offering For Elastic LB VM - Local Storage | 1 | 128 | 128 | Cloud.Com-ElasticLBVm-Local | 1 |
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
10 rows in set (0.00 sec)
```
New value
```
mysql> select id,name,cpu,speed,ram_size,unique_name,system_use from service_offering where name like "System%";
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
| id | name | cpu | speed | ram_size | unique_name | system_use |
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
| 3 | System Offering For Software Router | 1 | 500 | 512 | Cloud.Com-SoftwareRouter | 1 |
| 4 | System Offering For Software Router - Local Storage | 1 | 500 | 512 | Cloud.Com-SoftwareRouter-Local | 1 |
| 5 | System Offering For Internal LB VM | 1 | 256 | 512 | Cloud.Com-InternalLBVm | 1 |
| 6 | System Offering For Internal LB VM - Local Storage | 1 | 256 | 512 | Cloud.Com-InternalLBVm-Local | 1 |
| 7 | System Offering For Console Proxy | 1 | 500 | 1024 | Cloud.com-ConsoleProxy | 1 |
| 8 | System Offering For Console Proxy - Local Storage | 1 | 500 | 1024 | Cloud.com-ConsoleProxy-Local | 1 |
| 9 | System Offering For Secondary Storage VM | 1 | 500 | 512 | Cloud.com-SecondaryStorage | 1 |
| 10 | System Offering For Secondary Storage VM - Local Storage | 1 | 500 | 512 | Cloud.com-SecondaryStorage-Local | 1 |
| 11 | System Offering For Elastic LB VM | 1 | 128 | 512 | Cloud.Com-ElasticLBVm | 1 |
| 12 | System Offering For Elastic LB VM - Local Storage | 1 | 128 | 512 | Cloud.Com-ElasticLBVm-Local | 1 |
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
10 rows in set (0.01 sec)
```
* debian12: fix test_network_ipv6 and test_vpc_ipv6
* python3: remove duplicated imports
* debian12: failed to start Apache2 server (SSLCipherSuite @SECLEVEL=0)
error message
```
[Sat Jan 20 22:51:14.595143 2024] [ssl:emerg] [pid 10200:tid 140417063888768] AH02562: Failed to configure certificate cloudinternal.com:443:0 (with chain), check /etc/ssl/certs/cert_apache.crt
[Sat Jan 20 22:51:14.595234 2024] [ssl:emerg] [pid 10200:tid 140417063888768] SSL Library Error: error:0A00018E:SSL routines::ca md too weak
AH00016: Configuration Failed
```
openssl version
```
root@s-167-VM:~# openssl version -a
OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)
built on: Mon Oct 23 17:52:22 2023 UTC
platform: debian-amd64
options: bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-3.0.11=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-3"
MODULESDIR: "/usr/lib/x86_64-linux-gnu/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0x80202001478bfffd:0x0
```
certificate
```
root@s-167-VM:~# keytool -printcert -rfc -file /usr/local/cloud/systemvm/certs/realhostip.crt
-----BEGIN CERTIFICATE-----
MIIFZTCCBE2gAwIBAgIHKBCduBUoKDANBgkqhkiG9w0BAQUFADCByjELMAkGA1UE
BhMCVVMxEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxGjAY
BgNVBAoTEUdvRGFkZHkuY29tLCBJbmMuMTMwMQYDVQQLEypodHRwOi8vY2VydGlm
aWNhdGVzLmdvZGFkZHkuY29tL3JlcG9zaXRvcnkxMDAuBgNVBAMTJ0dvIERhZGR5
IFNlY3VyZSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTERMA8GA1UEBRMIMDc5Njky
ODcwHhcNMTIwMjAzMDMzMDQwWhcNMTcwMjA3MDUxMTIzWjBZMRkwFwYDVQQKDBAq
LnJlYWxob3N0aXAuY29tMSEwHwYDVQQLDBhEb21haW4gQ29udHJvbCBWYWxpZGF0
ZWQxGTAXBgNVBAMMECoucmVhbGhvc3RpcC5jb20wggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQCDT9AtEfs+s/I8QXp6rrCw0iNJ0+GgsybNHheU+JpL39LM
TZykCrZhZnyDvwdxCoOfE38Sa32baHKNds+y2SHnMNsOkw8OcNucHEBX1FIpOBGp
h9D6xC+umx9od6xMWETUv7j6h2u+WC3OhBM8fHCBqIiAol31/IkcqDxxsHlQ8S/o
CfTlXJUY6Yn628OA1XijKdRnadV0hZ829cv/PZKljjwQUTyrd0KHQeksBH+YAYSo
2JUl8ekNLsOi8/cPtfojnltzRI1GXi0ZONs8VnDzJ0a2gqZY+uxlz+CGbLnGnlN4
j9cBpE+MfUE+35Dq121sTpsSgF85Mz+pVhn2S633AgMBAAGjggG+MIIBujAPBgNV
HRMBAf8EBTADAQEAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAOBgNV
HQ8BAf8EBAMCBaAwMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5nb2RhZGR5
LmNvbS9nZHMxLTY0LmNybDBTBgNVHSAETDBKMEgGC2CGSAGG/W0BBxcBMDkwNwYI
KwYBBQUHAgEWK2h0dHA6Ly9jZXJ0aWZpY2F0ZXMuZ29kYWRkeS5jb20vcmVwb3Np
dG9yeS8wgYAGCCsGAQUFBwEBBHQwcjAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Au
Z29kYWRkeS5jb20vMEoGCCsGAQUFBzAChj5odHRwOi8vY2VydGlmaWNhdGVzLmdv
ZGFkZHkuY29tL3JlcG9zaXRvcnkvZ2RfaW50ZXJtZWRpYXRlLmNydDAfBgNVHSME
GDAWgBT9rGEyk2xF1uLuhV+auud2mWjM5zArBgNVHREEJDAighAqLnJlYWxob3N0
aXAuY29tgg5yZWFsaG9zdGlwLmNvbTAdBgNVHQ4EFgQUZyJz9/QLy5TWIIscTXID
E8Xk47YwDQYJKoZIhvcNAQEFBQADggEBAKiUV3KK16mP0NpS92fmQkCLqm+qUWyN
BfBVgf9/M5pcT8EiTZlS5nAtzAE/eRpBeR3ubLlaAogj4rdH7YYVJcDDLLoB2qM3
qeCHu8LFoblkb93UuFDWqRaVPmMlJRnhsRkL1oa2gM2hwQTkBDkP7w5FG1BELCgl
gZI2ij2yxjge6pOEwSyZCzzbCcg9pN+dNrYyGEtB4k+BBnPA3N4r14CWbk+uxjrQ
6j2Ip+b7wOc5IuMEMl8xwTyjuX3lsLbAZyFI9RCyofwA9NqIZ1GeB6Zd196rubQp
93cmBqGGjZUs3wMrGlm7xdjlX6GQ9UvmvkMub9+lL99A5W50QgCmFeI=
-----END CERTIFICATE-----
Warning:
The certificate uses the SHA1withRSA signature algorithm which is considered a security risk. This algorithm will be disabled in a future update.
```
it comes from
```
$ openssl x509 -in ./systemvm/agent/certs/realhostip.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 11277268652730408 (0x28109db8152828)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certificates.godaddy.com/repository, CN = Go Daddy Secure Certification Authority, serialNumber = 07969287
Validity
Not Before: Feb 3 03:30:40 2012 GMT
Not After : Feb 7 05:11:23 2017 GMT
Subject: O = *.realhostip.com, OU = Domain Control Validated, CN = *.realhostip.com
```
* debian12: use ed25519 instead of rsa as ssh-rsa has been deprecated in OpenSSH
on xenserver
```
[root@pr8497-t8906-xenserver-71-xs2 ~]# ssh -i .ssh/id_rsa.cloud -p 3922 169.254.214.153
Warning: Permanently added '[169.254.214.153]:3922' (ECDSA) to the list of known hosts.
Permission denied (publickey).
```
in the CPVM
Jan 22 19:31:09 v-1-VM sshd[2869]: userauth_pubkey: signature algorithm ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Jan 22 19:31:09 v-1-VM sshd[2869]: Connection closed by authenticating user root 169.254.0.1 port 54704 [preauth]
```
ssh-dss (DSA) is not supported either
* debian12: add PubkeyAcceptedAlgorithms=+ssh-rsa to sshd_config
* VR: install python3 packages in case of Debian 11
* pom.xml: exclude systemvm/agent/packages/* in license check
* systemvm: do not patch router/systemvm during startup
this will cause 4.19 SYSTEM template not work, but may be expected
- python3 VS python2 (default)
- openSSL 3.0.1 VS 1.1.1w
- openssh-server 9.1 VS 8.4
* VR: patch router/systemvm if template is debian11
This supports debian 11 template by
- revert change in systemvm/debian/etc/ssh/sshd_config
- patch VR/systemvms during startup
- install packages during patching system vm/routers
* python3 flake: fix E502 the backslash is redundant between brackets
```
../debian/root/health_checks/router_version_check.py:55:70: E502 the backslash is redundant between brackets
../debian/root/health_checks/router_version_check.py:58:61: E502 the backslash is redundant between brackets
../debian/root/health_checks/router_version_check.py:67:71: E502 the backslash is redundant between brackets
../debian/root/health_checks/router_version_check.py:70:60: E502 the backslash is redundant between brackets
../debian/root/health_checks/haproxy_check.py:47:71: E502 the backslash is redundant between brackets
../debian/root/health_checks/haproxy_check.py:48:64: E502 the backslash is redundant between brackets
../debian/root/health_checks/cpu_usage_check.py:43:54: E502 the backslash is redundant between brackets
../debian/root/health_checks/cpu_usage_check.py:46:58: E502 the backslash is redundant between brackets
../debian/root/health_checks/memory_usage_check.py:31:65: E502 the backslash is redundant between brackets
../debian/root/health_checks/memory_usage_check.py:42:57: E502 the backslash is redundant between brackets
../debian/root/health_checks/memory_usage_check.py:45:63: E502 the backslash is redundant between brackets
```
* python3 flake: fix E275 missing whitespace after keyword
```
../debian/opt/cloud/bin/cs_firewallrules.py:29:20: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_dhcp.py:27:16: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_dhcp.py:36:16: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_guestnetwork.py:33:20: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_guestnetwork.py:35:16: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_vpnusers.py:37:16: E275 missing whitespace after keyword
../debian/opt/cloud/bin/merge.py:230:11: E275 missing whitespace after keyword
../debian/opt/cloud/bin/merge.py:239:19: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_remoteaccessvpn.py:24:12: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_site2sitevpn.py:24:12: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs/CsHelper.py:90:15: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs/CsAddress.py:367:15: E275 missing whitespace after keyword
```
* python3 flake: fix configure.py
```
../debian/opt/cloud/bin/configure.py:24:22: E401 multiple imports on one line
../debian/opt/cloud/bin/configure.py:43:180: E501 line too long (294 > 179 characters)
../debian/opt/cloud/bin/configure.py:46:1: E302 expected 2 blank lines, found 1
../debian/opt/cloud/bin/configure.py:63:1: E302 expected 2 blank lines, found 1
../debian/opt/cloud/bin/configure.py:65:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/configure.py:72:1: E302 expected 2 blank lines, found 1
../debian/opt/cloud/bin/configure.py:310:25: E711 comparison to None should be 'if cond is not None:'
../debian/opt/cloud/bin/configure.py:312:29: E711 comparison to None should be 'if cond is None:'
../debian/opt/cloud/bin/configure.py:378:25: E711 comparison to None should be 'if cond is not None:'
../debian/opt/cloud/bin/configure.py:380:29: E711 comparison to None should be 'if cond is None:'
../debian/opt/cloud/bin/configure.py:490:29: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
../debian/opt/cloud/bin/configure.py:642:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/configure.py:644:18: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/configure.py:1416:1: E305 expected 2 blank lines after class or function definition, found 1
```
* python3 flake: fix other python files
```
../debian/opt/cloud/bin/vmdata.py:97:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/vmdata.py:99:14: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/cs/CsRedundant.py:438:53: E203 whitespace before ':'
../debian/opt/cloud/bin/cs/CsRedundant.py:461:53: E203 whitespace before ':'
../debian/opt/cloud/bin/cs/CsRedundant.py:499:5: E303 too many blank lines (2)
../debian/opt/cloud/bin/cs/CsDatabag.py:189:1: E302 expected 2 blank lines, found 1
../debian/opt/cloud/bin/cs/CsDatabag.py:193:37: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/cs/CsHelper.py:118:30: E231 missing whitespace after ','
../debian/opt/cloud/bin/cs/CsHelper.py:119:15: E225 missing whitespace around operator
../debian/opt/cloud/bin/cs/CsHelper.py:127:19: E225 missing whitespace around operator
../debian/opt/cloud/bin/cs/CsAddress.py:324:43: E221 multiple spaces before operator
../debian/opt/cloud/bin/cs/CsVpcGuestNetwork.py:28:1: E302 expected 2 blank lines, found 1
```
* python3 flake: fix CsNetfilter.py
```
../debian/opt/cloud/bin/cs/CsNetfilter.py:226:13: E117 over-indented
../debian/opt/cloud/bin/cs/CsNetfilter.py:233:180: E501 line too long (197 > 179 characters)
../debian/opt/cloud/bin/cs/CsNetfilter.py:241:14: E201 whitespace after '{'
../debian/opt/cloud/bin/cs/CsNetfilter.py:242:14: E201 whitespace after '{'
../debian/opt/cloud/bin/cs/CsNetfilter.py:247:18: E201 whitespace after '{'
../debian/opt/cloud/bin/cs/CsNetfilter.py:247:74: E202 whitespace before '}'
../debian/opt/cloud/bin/cs/CsNetfilter.py:248:18: E201 whitespace after '{'
```
* systemvm/test: fix sys.path
```
$ bash runtests.sh
/usr/bin/python
Python 3.10.12
Running pycodestyle to check systemvm/python code for errors
Running pylint to check systemvm/python code for errors
Python 3.10.12
pylint 2.12.2
astroid 2.9.3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
Running systemvm/python unit tests
....Device "eth0" does not exist.
.....................
----------------------------------------------------------------------
Ran 25 tests in 0.008s
OK
```
* Revert "systemvm template: remove hyperv packages and do not export"
This reverts commit 4383d59d031bde6eae7ebba261ff641ca0a66cd5.
* debian12: move SQL change to schema-41900to42000.sql
* debian12: update systemvm template version to 4.20 in pom.xml
* pom.xml: fix NPE if templates do not exist on download.cloudstack.org
* debian12: increase default system offering for routers to 384MiB RAM
* CKS: fix addkubernetessupportedversion failed with JRE17
```
marvin.cloudstackException.CloudstackAPIException: Execute cmd: addkubernetessupportedversion failed, due to: errorCode: 530, errorText:Cannot invoke "org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine$State.toString()" because the return value of "com.cloud.api.query.vo.TemplateJoinVO.getState()" is null
```
* python3: revert changes by 2to3 with systemvm/debian/root/health_checks/*.py
* debian12: use ISO/packages on download.cloudstack.org
* VR: Update default ram size to 384
* debian12: fix router_version_check.py after VR live-patch and add health check in test_routers.py
* debian12: fix build error after log4j 2.x merge
* VR: Update default ram size to 512MB (again)
This reverts commit 578dd2b73f380e8231ae1eb59827230757cac5e8 and efafa8c4d63775653a2cd406fca10784fbcec3e3.
* systemvmtemplate: Upgrade to Debian 12.5.0
* systemvm template: increase swap to 512MB
* VR: fix health check error due to deprecated SafeConfigParser
warning below
```
root@r-20-VM:~# /opt/cloud/bin/getRouterMonitorResults.sh true
/root/monitorServices.py:59: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in Python 3.12. Use ConfigParser directly instead.
parser = SafeConfigParser()
```
* test: fix wget does not work in macchinina vms on vmware80u1
fixes error below
```
{Cmd: wget -t 1 -T 1 www.google.com via Host: 10.0.55.186} {returns: ["wget: '/usr/lib/libpcre.so.1' is not an ELF file", "wget: can't load library 'libpcre.so.1'"]}
```
* packaging: add message for VR memory upgrade after packages installation
---------
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Vishesh <vishesh92@gmail.com>
|
||
|
Abhishek Kumar
|
592038a304
|
api,server,ui: granular resource limit management (#8362)
Feature spec: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Granular+Resource+Limit+Management Introduces the concept of tagged resource limits for granular resource limit management. Limits can be enforced on accounts and domains for the deployment of entities for a tagged resource. Current tagged resource limits can be used for the following resource types, Host limits - user_vm - cpu - memory Storage limits - volume - primary_storage Following global settings can used to specify tags for which limit needs to be enforced, Host: `resource.limit.host.tags` Storage: `resource.limit.storage.tags` Option for specifying tagged resource limits and viewing tagged resource usage are made available in the UI. Enhances the use of templatetag for VM deployment and template creation Adds option to list service/compute offerings that can be used with a given template. A new parameter named templateid has been added. Adds option to list disk offering with suitability flag for a virtual machine. A new parameter named virtualmachineid has been added to the listDiskOfferings API which when passed returns suitableforvirtualmachine param in the response. |
||
|
Wei Zhou
|
6af1c25f52 | Merge remote-tracking branch 'apache/4.19' | ||
GaOrtiga
|
6f3e4e6302
|
fix_filter_and_pagination (#8306)
Co-authored-by: Gabriel <gabriel.fernandes@scclouds.com.br> |
||
|
Abhishek Kumar
|
7dffbc6e47 |
Updating pom.xml version numbers for release 4.20.0.0-SNAPSHOT
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> |
||
|
Abhishek Kumar
|
a7b97ff3b0 |
Updating pom.xml version numbers for release 4.19.1.0-SNAPSHOT
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> |
||
|
Abhishek Kumar
|
2746225b99 |
Updating pom.xml version numbers for release 4.19.0.0
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> |
||
kishankavala
|
80bbb29abf
|
CleanUp Async Jobs after mgmt server maintenance (#8394)
This PR fixes moves resources stuck in transition state during async job cleanup Problem: During maintenance of the management server, other servers in the cluster or the same server after a restart initiate async job cleanup. However, this process leaves resources in a transitional state. The only recovery option currently available is to make direct database changes. Solution: This PR introduces a resolution by changing Volume, Virtual Machine, and Network resources from their transitional states. This adjustment enables the reattempt of failed operations without the need for manual database modifications. |
||
|
Abhishek Kumar
|
3936f7c2cf
|
vm-import: kvm import and fix volume size when lesser than 1GiB (#8500)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> Co-authored-by: Daan Hoogland <daan@onecht.net> |
||
|
Nicolas Vazquez
|
b8d3e342be
|
Fix KVM import unmanaged instances on basic zone (#8465)
This PR fixes import unmanaged instances on KVM basic zones, on top of #8433 Fixes: #8439: point 1 |
||
|
kishankavala
|
ab20b1220f
|
KVM Ingestion - Import Instance (#7976)
This PR adds new functionality to import KVM instances from an external host or from disk images in local or shared storage. Doc PR: https://github.com/apache/cloudstack-documentation/pull/356 |