2548 Commits

Author SHA1 Message Date
Suresh Kumar Anaparti
6d16ac2113
ScaleIO/PowerFlex smoke tests improvements, and some fixes (#11554)
* ScaleIO/PowerFlex smoke tests improvements, and some fixes

* Fix test_volumes.py, encrypted volume size check (for powerflex volumes)

* Fix test_over_provisioning.py (over provisioning supported for powerflex)

* Update vm snapshot tests

* Update volume size delta in primary storage resource count for user vm volumes only
The VR volumes resource count for PowerFlex volumes is updated here, resulting in resource count discrepancy
(which is re-calculated through ResourceCountCheckTask later, and skips the VR volumes)

* Fix test_import_unmanage_volumes.py (unsupported for powerflex)

* Fix test_sharedfs_lifecycle.py (volume size check for powerflex)

* Update powerflex.connect.on.demand config default to true
2025-09-12 16:17:20 +02:00
Wei Zhou
973b333e40
Merge remote-tracking branch 'apache/4.19' into 4.20 2025-09-02 08:24:20 +02:00
Wei Zhou
48648d4c45
test: fix test_04_rvpc_network_garbage_collector_nics failure (#11542)
* test: fix test_04_rvpc_network_garbage_collector_nics failure

* test: check backup VR as well
2025-09-02 08:08:10 +02:00
Daan Hoogland
609efcc231 Merge branch '4.19' into 4.20 2025-07-25 22:01:17 +02:00
ghernadi
a4263da8ae
linstor: Use template's uuid if pool's downloadPath is null as resource-name (#11053)
Also added an integration test for templates from snapshots
2025-07-25 07:51:11 -04:00
Pearl Dsilva
379ee07d88 Updating pom.xml version numbers for release 4.19.4.0-SNAPSHOT
Signed-off-by: Pearl Dsilva <pearl1594@gmail.com>
2025-06-06 18:00:09 +05:30
Pearl Dsilva
b5e2c181f9 Updating pom.xml version numbers for release 4.20.2.0-SNAPSHOT
Signed-off-by: Pearl Dsilva <pearl1594@gmail.com>
2025-06-06 15:38:12 +05:30
Pearl Dsilva
c61a5eb430 Updating pom.xml version numbers for release 4.20.1.0
Signed-off-by: Pearl Dsilva <pearl1594@gmail.com>
2025-05-30 12:43:00 +05:30
Daan Hoogland
0c7d47138d Updating pom.xml version numbers for release 4.19.3.0
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-05-30 09:08:58 +02:00
Rohit Yadav
f99fb3ecde Merge remote-tracking branch 'origin/4.19' into 4.20 2025-05-22 11:46:59 +05:30
Wei Zhou
4748c3fd4d
test: cleanup acl in test_global_acls.py (#10893) 2025-05-22 11:40:08 +05:30
Wei Zhou
538e35f8ad
test: fix test_restore_vm failure on vmware (#10885)
* test: fix test_restore_vm failure on vmware

* Fix URL for cloud-init vms on vmware/xen
2025-05-19 12:24:18 +02:00
Suresh Kumar Anaparti
90316b2e90
VMware 80u2 and 80u3 updates/fixes (#10586)
* VMware - Ignore disk not found error on cleanup when the VM disk doesn't exists

* VMware - Retry powerOn on lock issues

* addressed comments

* Update CPVM reboot tests - wait for the agent to Disconnect and back Up

* Retry moveDatastoreFile when any file access issue while creating volume from snapshot

* Update full clone flag when restoring vm using root disk offering with more size than the template size

* refactored (mainly,for diskInfo - causing NPE in some cases)

* Retry moveDatastoreFile when there is any file access issue
2025-05-17 00:39:34 +05:30
Pearl Dsilva
0648d000b2
Support XenServer 8.4 / XCP 8.3 - make scripts python3 compatible (#10684) 2025-05-13 12:35:04 +02:00
Pearl Dsilva
a21f912be3 Merge branch '4.19' of https://github.com/apache/cloudstack into 4.20 2025-05-12 12:41:34 +05:30
dahn
0a090f4853
cleanup call on super (#10807) 2025-05-09 12:54:32 +02:00
Wei Zhou
c45ed06098
test: fix test_hostha_simulator.py and test_outofbandmanagement.py (#10815) 2025-05-08 09:17:00 +02:00
Abhishek Kumar
c1923a2dcb
test: cleanup test_guest_os.py for multiple execution (#10818)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-05-07 08:50:36 +02:00
Pearl Dsilva
83c3ea2e67
smoke tests / CI : Fix test_vm_stric_host_tags (#10806) 2025-05-04 08:16:28 +02:00
Pearl Dsilva
0785ba046e Merge branch '4.19' of https://github.com/apache/cloudstack into 4.20 2025-04-28 11:10:08 +05:30
Pearl Dsilva
9d9737ae04
smoke tests: Fix cluster DRS & non-strict host affinity smoke test failures on XenServer / XCP-ng (#10761) 2025-04-26 17:39:17 +02:00
Pearl Dsilva
07f4fc2e51
Fix smoke tests due to change in behavior of restore VM (#10583)
* Fix unit tests due to change in behavior of restore VM

* update numbering in comments

* revert delete operations

* fix placement of start vm after refactoring
2025-04-26 10:11:27 +05:30
Pearl Dsilva
1adfaf90ad
Smoke tests: Xenserver - Fix consistent failure noticed on scale VM test (#10741)
* Smoke tests: Xenserver - Fix consistent failure noticed on scale VM test

* update test
2025-04-25 10:49:04 +02:00
Wei Zhou
55c8138a1a
test: fix test_certauthority_root.py (#10762)
it does not work with python3
```
2025-04-18T10:43:58.5235913Z 2025-04-18 10:32:20,503 - CRITICAL - EXCEPTION: Failure:: ['Traceback (most recent call last):\n', '  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/unittest/case.py", line 59, in testPartExecutor\n    yield\n', '  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/unittest/case.py", line 591, in run\n    self._callTestMethod(testMethod)\n', '  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/unittest/case.py", line 549, in _callTestMethod\n    method()\n', '  File "/home/runner/.local/lib/python3.10/site-packages/nose/failure.py", line 35, in runTest\n    raise self.exc_val.with_traceback(self.tb)\n', '  File "/home/runner/.local/lib/python3.10/site-packages/nose/loader.py", line 335, in loadTestsFromName\n    module = self.importer.importFromPath(\n', '  File "/home/runner/.local/lib/python3.10/site-packages/nose/importer.py", line 162, in importFromPath\n    return self.importFromDir(dir_path, fqname)\n', '  File "/home/runner/.local/lib/python3.10/site-packages/nose/importer.py", line 198, in importFromDir\n    mod = load_module(part_fqname, fh, filename, desc)\n', '  File "/home/runner/.local/lib/python3.10/site-packages/nose/importer.py", line 128, in load_module\n    spec.loader.exec_module(mod)\n', '  File "<frozen importlib._bootstrap_external>", line 883, in exec_module\n', '  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed\n', '  File "/home/runner/work/cloudstack/cloudstack/test/integration/smoke/test_certauthority_root.py", line 27, in <module>\n    from OpenSSL.crypto import FILETYPE_PEM, verify, X509\n', "ImportError: cannot import name 'verify' from 'OpenSSL.crypto' (unknown location)\n"]
```
2025-04-24 14:13:20 +05:30
dahn
5a5c865133
Revert "Add the option to filter by host when retrieving of unregistered VMs (#9925)" (#10647)
This reverts commit aa6c581e405d7652916a9290e042f0bd26c3913a.
2025-04-24 09:06:57 +02:00
dahn
6f334eb187
Update test_linstor_volumes.py for spello 2025-03-26 12:14:45 +01:00
Abhishek Kumar
35a7438033
test: fix failure in test_06_purge_expunged_vm_background_task (#10501)
Failures seen in https://github.com/apache/cloudstack/pull/10006#issuecomment-2691067265 and others.

With https://github.com/apache/cloudstack/pull/9773, the response of
listManagementServers API has been changed. serviceip has been renamed
to ipaddress. This was causing not all MS getting restarted and purge
b/g task not being able to run. The code handling API response in the
test has been updated.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-03-06 09:25:22 +05:30
Abhishek Kumar
cdf19b552c
lint: fix test_linstor_volumes.py (#10502)
Fix lint error in the file - test/integration/plugins/linstor/test_linstor_volumes.py

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-03-05 08:49:47 -05:00
Pearl Dsilva
48ed5e2417 Merge branch '4.19' into 4.20 2025-03-04 09:17:16 -05:00
Rene Peinthor
217e534446
linstor: improve integration-tests (#10439)
make tests easier to run and without modifying the test to match the
correct system. Also add tests for volume and vm instance snapshots.
Improve runtime by ~6 minutes.
2025-03-03 11:33:28 -05:00
Daan Hoogland
4a3686297d Updating pom.xml version numbers for release 4.19.3.0-SNAPSHOT
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-02-25 10:43:11 +01:00
Daan Hoogland
4e321d4356 Updating pom.xml version numbers for release 4.19.2.0
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-02-20 09:32:07 +01:00
Daan Hoogland
4f3e8e8c5a Merge branch '4.19' into 4.20 2025-02-12 15:00:51 +01:00
dahn
aa6c581e40
Add the option to filter by host when retrieving of unregistered VMs (#9925)
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
2025-02-10 17:06:39 +01:00
Abhishek Kumar
0b5a5e8043
api,agent,server,engine-schema: scalability improvements (#9840)
* api,agent,server,engine-schema: scalability improvements

Following changes and improvements have been added:

- Improvements in handling of PingRoutingCommand

    1. Added global config - `vm.sync.power.state.transitioning`, default value: true, to control syncing of power states for transitioning VMs. This can be set to false to prevent computation of transitioning state VMs.
    2. Improved VirtualMachinePowerStateSync to allow power state sync for host VMs in a batch
    3. Optimized scanning stalled VMs

- Added option to set worker threads for capacity calculation using config - `capacity.calculate.workers`

- Added caching framework based on Caffeine in-memory caching library, https://github.com/ben-manes/caffeine

- Added caching for account/use role API access with expiration after write can be configured using config - `dynamic.apichecker.cache.period`. If set to zero then there will be no caching. Default is 0.

- Added caching for account/use role API access with expiration after write set to 60 seconds.

- Added caching for some recurring DB retrievals

    1. CapacityManager - listing service offerings - beneficial in host capacity calculation
    2. LibvirtServerDiscoverer existing host for the cluster - beneficial for host joins
    3. DownloadListener - hypervisors for zone - beneficial for host joins
    5. VirtualMachineManagerImpl - VMs in progress- beneficial for processing stalled VMs during PingRoutingCommands

- Optimized MS list retrieval for agent connect

- Optimize finding ready systemvm template for zone

- Database retrieval optimisations - fix and refactor for cases where only IDs or counts are used mainly for hosts and other infra entities. Also similar cases for VMs and other entities related to host concerning background tasks

- Changes in agent-agentmanager connection with NIO client-server classes

    1. Optimized the use of the executor service
    2. Refactore Agent class to better handle connections.
    3. Do SSL handshakes within worker threads
    5. Added global configs to control the behaviour depending on the infra. SSL handshake could be a bottleneck during agent connections. Configs - `agent.ssl.handshake.min.workers` and `agent.ssl.handshake.max.workers` can be used to control number of new connections management server handles at a time. `agent.ssl.handshake.timeout` can be used to set number of seconds after which SSL handshake times out at MS end.
    6. On agent side backoff and sslhandshake timeout can be controlled by agent properties. `backoff.seconds` and `ssl.handshake.timeout` properties can be used.

- Improvements in StatsCollection - minimize DB retrievals.

- Improvements in DeploymentPlanner allow for the retrieval of only desired host fields and fewer retrievals.

- Improvements in hosts connection for a storage pool. Added config - `storage.pool.host.connect.workers` to control the number of worker threads that can be used to connect hosts to a storage pool. Worker thread approach is followed currently only for NFS and ScaleIO pools.

- Minor improvements in resource limit calculations wrt DB retrievals

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* test1, domaindetails, capacitymanager fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test2 - agent tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* capacitymanagertest fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* change

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix missing changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert marvin/setup.py

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix indent

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* use space in sql

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address duplicate

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* update host logs

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert e36c6a5d07

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix npe in capacity calculation

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* move schema changes to 4.20.1 upgrade

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix build

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add some more tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* checkstyle fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove unnecessary mocks

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* replace statics

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* engine/orchestration,utils: limit number of concurrent new agent
connections

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor - remove unused

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* unregister closed connections, monitor & cleanup

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add check for outdated vm filter in power sync

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* agent: synchronize sendRequest wait

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2025-02-01 12:28:41 +05:30
Harikrishna
9bc283e5c2
Introducing granular command timeouts global setting (#9659)
* Introducing granular command timeouts global setting

* fix marvin tests

* Fixed log messages

* some more log message fix

* Fix empty value setting

* Converted the global setting to non-dynamic

* set wait on command only when granular wait is defined. This is to keep the backward compatibility

* Improve error logging
2025-01-07 17:06:32 +05:30
João Jandre
c63c7ee63e Updating pom.xml version numbers for release 4.20.1.0-SNAPSHOT
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-27 11:40:45 -03:00
João Jandre
2fe3fcef7c Updating pom.xml version numbers for release 4.20.0.0
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-19 08:54:07 -03:00
Daan Hoogland
dbfc7f23a7 Merge branch '4.19' 2024-10-11 17:59:46 +02:00
Wei Zhou
5ab0a52d66 util: check JSESSIONID in cookies if user is passed 2024-10-11 17:24:34 +02:00
Daan Hoogland
2fa17617ba Merge release branch 4.19 to main
* 4.19:
  Fix ISO url in test_usage.py (#9739)
2024-09-30 11:58:03 +02:00
Wei Zhou
67ce326a8e
Fix ISO url in test_usage.py (#9739) 2024-09-28 08:54:45 +02:00
Vishesh
1303a4f323
Feature: Allow adding delete protection for VMs & volumes (#9633)
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-09-09 14:44:50 +02:00
Wei Zhou
679ce1a639
feature: Dynamic and Static Routing (#9470)
This PR contains 3 features

- IPv4 Static Routing (Routed mode) #9346
Design document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=306153967

- AS Numbers Management #9410
Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/BGP+AS+Numbers+Management


- Dynamic routing
Design Document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=315492858

- Document: https://github.com/apache/cloudstack-documentation/pull/419

Rename nsx mode to routing mode

by
```
git grep -l nsx_mode  |xargs sed -i "s/nsx_mode/routing_mode/g"
git grep -l nsxmode  |xargs sed -i "s/nsxmode/routingmode/g"
git grep -l nsxMode  |xargs sed -i "s/nsxMode/routingMode/g"
git grep -l NsxMode  |xargs sed -i "s/NsxMode/RoutingMode/g"
```
- re-organize sql changes
- fix NPE as rules do not have public ip
- fix missing destination cidr in ingress rules
- disable network usage for routed network
- fix DB exception as network_id is -1 during network creation
- apply ingress/egress routing rules
- VR changes to configure nft rules for isolated network
- VR: setup nft rule for control network
- VR: flush all iptables rules
- fix NPE which is because ingress rules do not have public ip associated
- fix dest cidr is missing in nft tables
- add ip4 routing and ip4 routes to list network and list vpc response
- fix ingress rule is missing when vr is restarted
- fix icmp types in nft rules
- add tab to manage routing firewall rules
- fix ingress rules are not applied when VR is restarted
- add default rules in FORWARD chain
- fix create vpc offerings
- fix public ip is not assigned to vpc
- fix network offering is not listed when create vpc tier
- add is_routing to boot args of vpc vr
- remove table ip4_firewall in vpc vr
- release or remove subnet when remove a network
- implemenent fw_vpcrouter_routing
- fix wrong ip familty when flush ipv4 rules
- fix acl rules are not applied due to wrong version (should be 6 which means ip6 rules are removed)
- add default rules for vpc tiers so that tcp connections (e.g. ssh) work
- append policy rules after default rules
- remove /usr/local/cloud/systemvm/ in routers
- throw an exception when allocate subnet with cidrsize
- fix some TODOs
- add new parameters to update API
- return type Ipv4GuestSubnetNetworkMap when get or create subnet
- fix firewall rules are broken
- add domain_id and account_id to db
- add domain/account/project to ipv4 subnet response
- create ipv4 subnet for domain/account/project
- check conflict when update ipv4 subnet
- ui changes
- add parent subnet to response
- add list for ipv4 subnet
- implement some methods
- fix list subnets for guest networks by zoneid
- UI changes
- fix delete ipv4 subnet for network
- fix ipv4 subnet is set to zone guest network cidr if cidrsize is specified
- add zone info to response if parent subnet is null but network is not
- fix gateway/cidr is not set when create network with cidrsize
- fix order of nft rules in the VRs

* Routed v24

- add classes in marvin base.py

* Routed v25

- add test_01_subnet_zone
- fix dedicate to domain/account failure
- list subnets for network by keyword and subnet

* Routed v26: implement subnet auto-allocation

- add utils for split ip ranges into small subnets
- add utils to get start/end ip of a cidr
- implement subnet auto-generation
- add global settings

* Routed 27: add subnet for VPC

- add db column for vpc_id
- add db record for vpc
- remove db record when delete a vpc
- add checkConflicts methods
- remove duplicated settings
- check ipv4 cidr when create subnet

* Routed v28: update smoke tests

- update test_ipv4_routing.py
- search subnets by networkid

* Routed 29: fix vpc and add more tests

- fix createnetwork in vpc
- add vpc id/name to response
- fix zone id/name are not displayed in some cases
- add smoke test for vpc
- add smoke tests for failed cases
- add smoke test for connectivity checks
- marvin: add "-q" to ssh command

* Routed 31: ui and smoke tests

- UI: add link to network in list view
- add nftables rules check in VRs

* Routed 32: add chain OUTPUT and more rules

- fix the issue 80/443/8080 is not reachable from VR itself

```
2024-06-27 10:21:52,121 INFO     Executing: systemctl start cloud-password-server@172.31.1.1
2024-06-27 10:21:52,128 INFO     Service cloud-password-server@172.31.1.1 start
2024-06-27 10:21:52,129 INFO     Executing: ps aux
2024-06-27 10:24:02,175 ERROR    Failed to update password server due to: <urlopen error [Errno 110] Connection timed out>
```

* Routed: fix dns search from VMs in Isolated networks

* Routed: fix VPC dns issue due to gateway IP is missing in cloud.conf

This is caused by NSX integration, and fixed by
https://github.com/apache/cloudstack/pull/9102/

* Routed: rename routing_mode to network_mode

* Routed: replace centos5.5 template in smoke test as dhclient does not work in the vms

// this does not work
refer to https://dominikrys.com/posts/disable-udp-checksum-validation/#ignoring-udp-checksums-with-nftables
and
https://forum.openwrt.org/t/udp-checksum-with-nftables/161522/11

the vm should have checksum offloading disabled

* Routed: fix smoke test due to wrong cidrlist of egress rules and missing ingress rule from VR

* PR 9346: fix lint error schema-41910to42000.sql

* PR 9346: ui polish v1

* PR 9346: create VPC with cidrsize

* Routed: fix test failures with test_network_ipv6 and test_vpc_ipv6 due to 'ssh -q'

* Routed: fix /usr/local/cloud/systemvm/ are removed after SSVM/CPVM reboot

* Routed: fix IP of additional nics of VPC VR is not gateway

* PR 9346: fix cidrsize check when create VPC with cidrsize

* Routed: fix test/integration/smoke/test_ipv4_routing.py:279:16: E713 test for membership should be 'not in'

* PR9346: fix/Update api

* PR 9346: set response object name

* PR9346: UI refactor and small fixes

* PR9346: change return type of getNetworkMode

* PR9346: move IPv4 subnet to seperated tab

* PR9346: revert IpRangesTabGuest.vue back to original

* PR9346: fix remove ipv4 subnet on UI

* PR9346: fix test_ipv4_routing.py

* AS Number Range Management

* Create AS Number Range for a Zone

* Fix build

* Add ListASNRange and fix create ASN range

* Add List AS numbers

* Add UI for AS Numbers

* Fix UI and filter AS Numbers

* Add AS Number on Isolated network creation and refactor UI and response

* Release AS Number

* Add network offering new columns

* Add UI support to view and add AS number and configure network offering

* Automatically assign AS Number if not specify AS number

* update variable name

* Fix routing mode check

* UI: Only allow selecting AS number when routing mode is Dynamic and specifyAsNumber is true

* UI: Only pass AS number when supported by the network offering

* Release AS number on network deletion

* Add deleteASNRange command (#81)

* API: List ASNumbers by asnumber (#83)

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* AS number management extensions

* Support AS number on VPC tier creation based on the offering

* Fix delete AS Range

* Fix UI values

* UI: Minor fix for releasing AS number

* UI: Move management of AS Range to Zone details view

* Fix specify_as_number column in network_offering table to set the default false

* Add events for AS number operations

* Allow users to list AS Numbers and fix network form for Normal users

* Add AS number details to list networks response

* Fix Allocated time format

* Fix Allocated time format

* support in details view too

* Fix: Do not release AS number if acquired network requires AS number

* Fix: Do not release AS number if acquired network requires AS number

* Fix typo

* Fix allocated release

* Fix event type

* UI: Add Routing mode and Specify AS to the network offering details

* UI: Add Routing mode and Specify AS to the network offering details

* Address comment

* Fix release AS number of network deletion

* Fix release AS number of network deletion

* Fix

* Restore release to its place based on the boolean

* Rename boolean

* API: Add networkId as listASNumber parameter

* Add Network name to the search view filter for AS numbers

* Present allocated time in human readable format - Pubilc IP / AS Numbers

* Add account / domain filter for AS numbers

* Add support for AS numbers on VPC offerings

* Refactor AS number allocation to VPC and non VPC isolated networks

* Checkstyle

* Add support for AS numbers on VPC offerings

* extend vpc offering view and vpcoffering response

* merge https://github.com/shapeblue/cloudstack-playtika/pull/115 and change network_id of as_numbers to include vpc_id

* Display AS number of VPC tiers as the AS number of the VPC

* extend asnumber response and ui support

* improve UI and as number response to view VPC details

* List only dynamic offerings for vpc tiers with specify as numbers

* Fix release AS number

* Fix AS number displayed as 0 when no AS number assigned

* Fix VPC offering creation without specify AS

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix release AS number on VPC deletion

* Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java

* Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java

* Fix missing column on asnumber table

* Fix listASNumbers API to support vpcid and obtain AS number from vpc for tiers

* Prevent listing 0 AS number for VPC

* Fix create Isolated Network form

* Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

* Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

* Dynamic: move routingmode/specifyasn after networkmode in AddNetworkOffering.vue on UI

* Dynamic: fix ip4routing in network response

* Dynamic/systemvm: add FRR to systemvm template

* Dynamic: BGP peers (DB,VO,Dao)

* Dynamic: BGP peers (VR/server)

* Dynamic: v3

- remove BgpPeer class
- fix vpc vr has bgp peers of only 1 tier
- rename ip4_cidr to guest_ip4_cidr
- rename ip6_cidr to guest_ip6_cidr
- generate /etc/frr/frr.conf
- apply BGP peers on Dynamic-Routed network even if there is no BGP peers

* Dynamic v4: fix vpc vr

- fix duplicated guest cidr in frr.conf in vpc vr

todo
- restart frr / reload frr (reload will cause bgp session to Policy state)
- apis for bgp peers
- assign/release bgp peer from/to network

* Dynamic v5: add apis for bgp peers

* Dynamic v6: fix bugs

- set response object name
- remove required as number when update
- fix checks when update
- allow regular users to list bgp peers

* Dynamic v7: move apis to bgp sub-dir

* Dynamic v8: add tab for manage BGP peers on UI

* Dynamic v9: fix update bgp with same config

* Dynamiv v10: add changeBgpPeersForNetworkCmd

* Dynamic v11: create network with bgppeerids

- create network with bgppeerids
- add marvin classes
- add smoke tests
- remove uuid from bgp_peer_network_map
- fix created/removed in bgp_peer_network_map
- remove bgppeers when remove a network
- UI: fix delete bgp peer

* Dynamic v12: add test for vpc tiers

* Dynamic v13: bug fixes

- fix change BGP peers for network in Allocated state
- fix listing network returns removed record
- fix all vpc tiers have the same settings
- remove BGP peers as part of network removal
- remove FRR settings for vpc tiers without any BGP peers
- UI: fix no error msg when change BGP peers

* Dynamic v14: assign BGP Peers for VPC instead of VPC tiers

- create vpc with bgppeerids
- do not allow create/update vpc tier with bgppeerids
- apply all bgp peers when create/delete a vpc tier
- UI: change bgp peers for vpc
- test: update tests on vpc

* Dynamic: fix build errors after merging as number PR

* Dynamic: fix TODOs

* Dynamic: fix smoke test on VPC

* Allow creation of networks by users with as numbers

* Address review comments

* Move BGPService to bgp package and inject it on BaseCmd

* Revert changes for CKS and address more comments

* Display left side menu option for AS number only for root admin

* Dynamic: create/update BGP peer with details

refer to https://docs.frrouting.org/en/latest/bgp.html

* Dynamic: fix build error and remove access to ListBgpPeers cmd for regular users

* Dynamic: assign all zone BGP peers to user networks

* Dynamic: show BGP peer info of networks only for root admin

* AS number: disable specifyasnumber for non-NSX offerings

* Dynamic: pass bgppeer details to command and fix typo with ip6 addr

* Dynamic: list BGP peers by isdedicated, and fix change bgppeers for network/vpc

* Dynamic: add UI labels

* Dynamic: add bgp peers to vpc response

* Dynamic: list bgp peers by keyword, fix list by asnumber

* Dynamic: fix list bgppeers by keyword and db schema

* Dynamic: fix list bgppeers do not return dedicated peers

* Dynamic: update UI when create network/vpc offering

* Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* Update tools/marvin/setup.py

* Dynamic: network mode must be same when update a network with new offering

* Dynamic: add method networkModel.isAnyServiceSupportedInNetwork

* Dynamic: rename APIs and classes

* Dynamic: fix unit tests due to previous changes

* Dynamic: validateNetworkCidrSize when auto-create subnet

* Dynamic: check AS number overlap

* Dynamic: add ActionEvent

* Dynamic: small code optimization

* Dynamic: fix ui bugs after api rename

* Dynamic: add marvin and test for ASN ranges and AS numbers

* Dynamic: add account setting use.system.bgp.peers

also
- change the default value of routed.ipv4.vpc.max.cidr.size and routed.ipv4.vpc.min.cidr.size
- change the category of settings

* static: fix ui error when delete zone ipv4 subnets

* static: small UI polish

* Dynamic: throw exception when as number is required but not passed

* Dynamic: fix typo when create FRR directory which causes network deletion failures

* Dynamic: connect to ALL (or ALL dedicated) BGP peers if no BGP peer mapping for the network/vpc

* Dynamic: throw exception when as number is required for VPC but not passed

* Dynamic: list bgp peers by useSystemBgpPeers

* Dynamic: fix frr config in VPC VR when change bgp peers

* Dynamic: create frr config even if there is no VPC tiers

* Dynamic: list bgp peers by zoneid (required for account) and account

* Dynamic: only apply FRR config for vpc tiers with dynamic routing

* Dynamic: donot send commands to router if commands size is 0

* Dynamic: fix 'new IPv6 address is not valid' when update bgp peer without IPv6

* Dynamic: throw exception if fail to allocate AS number when create network/vpc with dynamic routing

* Dynamic: enable ipv6 unicast and 'ip nht resolve-via-default'

* Dynamic: delete network/vpc if fail to allocate AS number when create network/vpc with dynamic routing

* test: add unit tests for ASN APIs

* test: add unit tests for core module

* test: add unit tests for API responses

* test: add unit tests for BgpPeerTO

* test: add minor changes

* test: add tests for create/delete/update/list RoutingFirewallRuleCmd

* Static: show ip4 routes for vpc tiers

* test: fix smoke test failure caused by type change of as number

* test: add test for Ipv4SubnetForZoneCmd

* test: add test for Ipv4SubnetForGuestNetworkCmd and BgpPeerCmd

* UI: do not show redundant router when network mode is ROUTED as RVR is not supported

* UI: hide 'Conserve mode' when networkmode is ROUTED

* test: add unit tests for ListASNumbersCmdTest

* Static: remove allocated IPv4 subnet when delete a network or vpc

* test: add unit tests for BgpPeersRules

* Dynamic: set ipv4routing from network offering

* server: list as numbers and ipv4 subnets by keyword

* server: remove dedicated bgp peers and ipv4 subnets when delete an account or domain

* server: fix dedicated ipv4 subnet is allocated to other accounts

* UI: fix allocated time format

* server: ignore project is projectid is -1 so bgppeers/ipv4subnets works in project view

* UI: add project column to bgp peers and ipv4 subnets

* server: fix list AS numbers by domain admin or normal user

* server: fix network creation when ipv4 subnet is dedicated

* UI: polish network.js

* Dynamic: fix frr config for ipv6 routing

* Static routing: support cks cluster

* Static: get/create IPv4 subnet from dedicated subnets at first

* Dynamic: add BGP peers tab

* Static: remove redundant loops

* api: add since to api and response

* server: add unit tests

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-06 08:55:17 +05:30
Abhisar Sinha
605534b417
feature: Shared Storage Filesystem as a First Class Feature (#9208)
This PR implements Storage filesystem as a first class feature.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Filesystem+as+a+First+Class+Feature

Documentation PR: apache/cloudstack-documentation#420

Co-authored-by: Wei Zhou <weizhou@apache.org>
2024-09-05 17:22:32 +05:30
Vishesh
72d0546d8b
Shared Network Firewall (Security groups) in Advanced zone without security groups (#9415) 2024-09-05 10:35:05 +02:00
slavkap
12d9c26747
Added support for storpool_qos service (#8755) 2024-08-29 09:23:25 +02:00
Wei Zhou
f5efe56fb5
Merge remote-tracking branch 'apache/4.19' 2024-08-26 17:34:15 +02:00
Vishesh
bc28665679
Add support for network data in Config Drive (#9329) 2024-08-26 10:53:42 +02:00
Wei Zhou
0597def630
test: fix component tests test_acl_isolatednetwork and test_acl_isolatednetwork_delete (#9571)
* test: fix component tests test_acl_isolatednetwork and test_acl_isolatednetwork_delete

* Update test/integration/component/test_acl_isolatednetwork.py
2024-08-23 19:27:11 +02:00