37327 Commits

Author SHA1 Message Date
Rohit Yadav
5c365fa820
systemvmtemplate: bump new systemvmtemplate 4.19.1 (#9349)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-07-10 10:01:08 +05:30
John Bampton
c6d3f6e623
pre-commit: enable Python flake8 rule E271 (#9358) 2024-07-09 10:17:53 +02:00
John Bampton
a1d8bab43f
Bump actions/cache to v4 (#9353)
https://github.com/actions/cache
2024-07-09 13:47:11 +05:30
Wei Zhou
354627a19d
UI: fix CreateIsolatedNetworkForm.vue (#9359) 2024-07-09 13:46:28 +05:30
Suresh Kumar Anaparti
37c91abd3d
NPE fix, for test_hostha_kvm_host_fencing (#9355) 2024-07-09 12:20:10 +05:30
Vishesh
0af88353fb
Merge branch '4.19' 2024-07-09 10:40:21 +05:30
Henrique Sato
ca66b76cb9
Fix instance & backup resource table columns (#9147)
Co-authored-by: Henrique Sato <henrique.sato@scclouds.com.br>
2024-07-09 10:23:31 +05:30
Rohit Yadav
7977d1475e
plugins: make default signature check mandatory (#9357)
This improves upon #9219, to make the signature checks mandatory by
default but allows for users to relax the setting if they really must.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-07-09 09:49:27 +05:30
Wei Zhou
f381763917
test: fix simulator CI failure test_cluster_drs (#9354) 2024-07-09 01:18:50 +05:30
Abhishek Kumar
c35ba606a1
ui: fix account list on domain change in filterview (#9345)
* ui: fix account list on domain change in filterview

Fixes #9274

* fix

* filter accounts for selected domain

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-07-08 17:24:38 +05:30
John Bampton
b69cc0272d
pre-commit: enable Python Flake8 rule E713 (#9016)
Co-authored-by: Vishesh <vishesh92@gmail.com>
2024-07-08 09:00:18 +02:00
Pearl Dsilva
f35f3f6a96
UI: create network form constantly tries to list domains - which is not permitted for normal users (#9348) 2024-07-08 08:28:54 +02:00
Abhishek Kumar
1144f5258d Merge remote-tracking branch 'apache/4.19' 2024-07-08 11:36:44 +05:30
Abhishek Kumar
b080215c08 utils: honor custom timeout for script execution for exit value
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-07-08 11:35:28 +05:30
Abhishek Kumar
3c9bc07d06 Merge remote-tracking branch 'apache/4.19' 2024-07-06 13:01:50 +05:30
Abhishek Kumar
d99cf93d7c Merge remote-tracking branch 'apache/4.18' into 4.19 2024-07-06 11:31:23 +05:30
Abhishek Kumar
2cf838cc1d framework/cluster: improve cluster service, integration API server
- mTLS implementation for cluster service communication
- Listen only on the specified cluster node IP address instead of all interfaces
- Validate incoming cluster service requests are from peer management servers based on the server's certificate dns name which can be through global config - ca.framework.cert.management.custom.san
- Hardening of KVM command wrapper script exeicution
- Improve API server integration port check
- cloudstack-management.default: don't have JMX configuration if not needed. JMX is used for instrumentation; users who need to use it should enable it explicitly

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-07-06 11:29:26 +05:30
Abhishek Kumar
beebeed5e2 framework/cluster: improve cluster service and integration API service
- mTLS implementation for cluster service communication
- Listen only on the specified cluster node IP address instead of all interfaces
- Validate incoming cluster service requests are from peer management servers based on the server's certificate dns name which can be through global config - ca.framework.cert.management.custom.san
- Hardening of KVM command wrapper script execution
- Improve API server integration port check
- cloudstack-management.default: don't have JMX configuration if not needed. JMX is used for instrumentation; users who need to use it should enable it explicitly

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-07-05 19:05:06 +05:30
Vishesh
ba85b4d4ff
Merge release branch 4.18 to 4.19
* 4.18:
  linstor: deleteAsync fallback to volume UUID if path not set yet (#9325)
  linstor: do not use the same static ApiClient for all calls (#9326)
2024-07-05 12:45:27 +05:30
Rene Peinthor
eb1a0ef7f2
linstor: deleteAsync fallback to volume UUID if path not set yet (#9325) 2024-07-04 17:35:01 +05:30
Wei Zhou
374ea7d482
UI: fix "Request failed." error when open vnf appliances tab of network (#9334) 2024-07-04 17:32:52 +05:30
Abhishek Kumar
a35d79a681
ui: fix minor issue with webhook forms (#9335)
- Fix width of Test webhook delivery form
- Fix auto execution of test delivery in create webhook form

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-07-04 17:31:34 +05:30
Abhishek Kumar
ef5b5bbd4e Updating pom.xml version numbers for release 4.18.2.1
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
4.18.2.1
2024-07-04 16:16:56 +05:30
Abhishek Kumar
4f5561937c framework/cluster: improve cluster service and integration API service
- mTLS implementation for cluster service communication
- Listen only on the specified cluster node IP address instead of all interfaces
- Validate incoming cluster service requests are from peer management servers based on the server's certificate dns name which can be through global config - ca.framework.cert.management.custom.san
- Hardening of KVM command wrapper script execution
- Improve API server integration port check
- cloudstack-management.default: don't have JMX configuration if not needed. JMX is used for instrumentation; users who need to use it should enable it explicitly

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-07-04 16:08:18 +05:30
Rene Peinthor
94e5ceff7f
linstor-ui: Fix add primary storage if Linstor protocol selected (#9327)
url param wasn't set if protocol Linstor is selected
2024-07-04 15:03:38 +05:30
Abhisar Sinha
79b1427d3e
test_primary_storage_scope.py should only run with kvm, vmware and simulator (#9323)
* test_primary_storage_scope should only run with kvm, vmware and simulator

* move cluster create and storage pool create from setup to test so that they are cleaned up in case of failure

* fixed lint failure

* using super class' tearDown
2024-07-04 13:29:05 +05:30
Vishesh
7c32bd2506
Fixup main build errors (#9330)
* Fixup main build errors

* Fixup flaky test

* Address comments
2024-07-04 13:00:37 +05:30
Rene Peinthor
121a35d666
linstor: do not use the same static ApiClient for all calls (#9326)
The client.setBasePath() would overwrite the Linstor controller IP/host
for all current client users. This is basically a race condition
that triggered as soon as you had configured 2 different primary storages
with different Linstor controllers.
2024-07-03 16:13:16 +05:30
Gabriel Pordeus Santos
de683a5163
add ownership selection to create volume (#9265) 2024-07-01 10:39:36 +02:00
Vishesh
0ec7c72875
Merge branch '4.19' 2024-07-01 12:41:45 +05:30
Vishesh
e010c9be93
Fixup main build error (#9314) 2024-06-29 14:25:53 +05:30
Henrique Sato
d79735606b
Handle public IP race conditions (#9234)
* Lock public IP

* Release IP if ID is not null

* Fix NPEs

Co-authored-by: Henrique Sato <henrique.sato@scclouds.com.br>
2024-06-29 10:28:01 +05:30
Abhisar Sinha
063dc60114
Change storage pool scope from Cluster to Zone and vise versa (#8875)
* New feature: Change storage pool scope

* Added checks for Ceph/RBD

* Update op_host_capacity table on primary storage scope change

* Storage pool scope change integration test

* pull 8875 : Addressed review comments

* Pull 8875: remove storage checks, AbstractPrimayStorageLifeCycleImpl class

* Pull 8875: Fixed integration test failure

* Pull 8875: Review comments

* Pull 8875: review comments + broke changeStoragePoolScope into smaller functions

* Added UT for changeStoragePoolScope

* Rename AbstractPrimaryDataStoreLifeCycleImpl to BasePrimaryDataStoreLifeCycleImpl

* Pull 8875: Dao review comments

* Pull 8875: Rename changeStoragePoolScope.vue to ChangeStoragePoolScope.vue

* Pull 8875: Created a new smokes test file + A single warning msg in ui

* Pull 8875: Added cleanup in test_primary_storage_scope.py

* Pull 8875: Type in en.json

* Pull 8875: cleanup array in test_primary_storage_scope.py

* Pull:8875 Removing extra whitespace at eof of StorageManagerImplTest

* Pull 8875: Added UT for PrimaryDataStoreHelper and BasePrimaryDataStoreLifeCycleImpl

* Pull 8875: Added license header

* Pull 8875: Fixed sql query for vmstates

* Pull 8875: Changed icon plus info on disabled mode in apidoc

* Pull 8875: Change scope should not work for local storage

* Pull 8875: Change scope completion event

* Pull 8875: Added api findAffectedVmsForStorageScopeChange

* Pull 8875: Added UT for findAffectedVmsForStorageScopeChange and removed listByPoolIdVMStatesNotInCluster

* Pull 8875: Review comments + Vm name in response

* Pull 8875: listByVmsNotInClusterUsingPool was returning duplicate VM entries because of multiple volumes in the VM satisfying the criteria

* Pull 8875: fixed listAffectedVmsForStorageScopeChange UT

* listAffectedVmsForStorageScopeChange should work if the pool is not disabled

* Fix listAffectedVmsForStorageScopeChangeTest UT

* Pull 8875: add volume.removed not null check in VmsNotInClusterUsingPool query

* Pull 8875: minor refactoring in changeStoragePoolScopeToCluster

* Update server/src/main/java/com/cloud/storage/StorageManagerImpl.java

* fix eof

* changeStoragePoolScopeToZone should connect pool to all Up hosts

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-29 10:03:34 +05:30
Suresh Kumar Anaparti
2ca1b474bd
PowerFlex/ScaleIO SDC client connection improvements (#9268)
* Mitigation for non-scalable Powerflex/ScaleIO clients
- Added ScaleIOSDCManager to manage SDC connections, checks clients limit, prepare and unprepare SDC on the hosts.
- Added commands for prepare and unprepare storage clients to prepare/start and stop SDC service respectively on the hosts.
- Introduced config 'storage.pool.connected.clients.limit' at storage level for client limits, currently support for Powerflex only.

* tests issue fixed

* refactor / improvements

* lock with powerflex systemid while checking connections limit

* updated powerflex systemid lock to hold till sdc preparation

* Added custom stats support for storage pool, through listStoragePools API

* code improvements, and unit tests

* unit tests fixes

* Update config 'storage.pool.connected.clients.limit' to dynamic, and some improvements

* Stop SDC on host after migration if no volumes mapped to host

* Wait for SDC to connect after scini service start, and some log improvements

* Do not throw exception (log it) when SDC is not connected while revoking access for the powerflex volume

* some log improvements
2024-06-29 10:01:50 +05:30
Vishesh
90fe1d5fdc
Merge branch '4.19' 2024-06-29 03:35:24 +05:30
GaOrtiga
814c8b6a12
differentiate between instalation ISO and attached ISO (#9146)
Co-authored-by: Gabriel <gabriel.fernandes@scclouds.com.br>
2024-06-28 21:01:22 +05:30
Vishesh
bcbf152a05
Merge branch '4.18' into 4.19 2024-06-28 20:14:21 +05:30
Vishesh
94503e687c
UI: improve ux and fix some minor bugs (#9307) 2024-06-28 19:24:33 +05:30
dahn
6575375537
extract host from websocket to add route (#8922)
* extract host from websocket to add route

* move method to NetUtils

* imports

* import
2024-06-28 19:02:19 +05:30
Suresh Kumar Anaparti
983f164c57
Fixed src datastore on copy check for PowerFlex/ScaleIO storage driver (#9310) 2024-06-28 18:46:06 +05:30
Abhisar Sinha
644f3a3f48
Add, Delete Storage Pool commands should be able execute on a host in maintenance (#9301)
* Restart agent when host comes out of maintenance

* Don't send CreateStoragePoolCommand to hosts in maintenance mode

* CreateStoragePoolCommand can run when host in maintenance. Reverted the change to restart agent when host was already up and in maintenance

* Reverted changes done to ResourceManagerImplTest
2024-06-28 18:18:08 +05:30
Abhishek Kumar
c7f1ba5b8e
server: refactor listNetworks api database retrievals (#9184)
* server: refactor listNetworks api database retrievals

* fixes

* remove unused methods

* imports

* fix empty searchcriteria issue

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-28 17:19:32 +05:30
Abhisar Sinha
646c894ec6
Fix for race when automatically assigning IP to Vms (#9240)
* Fix for race when automatically assigning IP to Vms

* code refactor
2024-06-28 17:11:16 +05:30
Vishesh
d4446ee09d
Revert listAll behaviour for templates (#9309) 2024-06-28 15:58:44 +05:30
GaOrtiga
a5c8bb3bfc
server: apply rules when VR of Domain VPC is recreated (#8354)
* apply rules when VR is recreated

* Apply suggestions from code review

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

Co-authored-by: Gabriel <gabriel.fernandes@scclouds.com.br>
2024-06-28 15:46:20 +05:30
Vishesh
a4e9d7f21a
Change vm.stats.remove.batch.size to delete.batch.query.size & allow delete of volume_stats in batches (#9283)
* Change vm.stats.remove.batch.size to delete.batch.query.size

* Add support for deletion of volume stats in batches

* Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update configkey description

* Address comments

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-28 15:32:49 +05:30
Suresh Kumar Anaparti
46f672563e
Improve migration of external VMware VMs into KVM cluster (#8815)
* Create/Export OVA file of the VM on external vCenter host, to temporary conversion location (NFS)

* Fixed ova issue on untar/extract ovf from ova file
"tar -xf" cmd on ova fails with "ovf: Not found in archive" while extracting ovf file

* Updated VMware to KVM instance migration using OVA

* Refactoring and cleanup

* test fixes

* Consider zone wide pools in the destination cluster for instance conversion

* Remove local storage pool support as temporary conversion location
- OVA export not possible as the pool is not accessible outside host, NFS pools are supported.

* cleanup unused code

* some improvements, and refactoring

* import nic unit tests

* vmware guru unit tests

* Separate clone VM and create template file for VMware migration
- Export OVA (of the cloned VM) to the conversion location takes time.
- Do any validations with cloned VM before creating the template (and fail early).
- Updated unit tests.

* Check conversion support on host before clone vm / create template on vmware (and fail early)

* minor code improvements

* Auto select the host with instance conversion capability

* Skip instance conversion supported response param for non-KVM hosts

* Show supported conversion hosts in the UI

* Skip persistence map update if network doesn't exist

* Added support to export OVA from KVM host, through ovftool (when installed in KVM host)

* Updated importvm api param 'usemsforovaexport' to 'forcemstodownloadvmfiles', to be generic

* Updated hardcoded UI messages with message labels

* Updated UI to support importvm api param - forcemstodownloadvmfiles

* Improved instance conversion support checks on ubuntu hosts, and for windows guest vms

* Use OVF template (VM disks and spec files) for instance conversion from VMware, instead of OVA file
 - this would further increase the migration performance (as it reduces the time for OVA preparation / archiving of the VM files into a single file)

* OVF export tool parallel threads code improvements

* Updated 'convert.vmware.instance.to.kvm.timeout' config default value to 3 hrs

* Config values check & code improvements

* Updated import log, with time taken and vm details

* Support for parallel downloads of VMware VM disk files while exporting OVF from MS, and other changes below.
- Skip clone for powered off VMs
- Fixes to support standalone host (with its default datacenter)
- Some code improvements

* rebase fixes

* rebase fixes

* minor improvement

* code improvements - threads configuration, and api parameter changes to import vm files

* typo fix in error msg
2024-06-27 21:14:13 +05:30
Nima B
23f8856c7d
Update AssignToLoadBalancerRuleCmd.java (#9306)
changed vmidipmap[0].ip to vmidipmap[0].vmip
2024-06-27 19:50:36 +05:30
Abhishek Kumar
53faf0f66a
xenserver: attach regular iso with configdrive (#9216)
* xenserver: attach regular iso with configdrive

Fixes #7902

This PR allows attaching a regular ISO to a VM when it already has the
config drive ISO attached.
Config-drive ISO is now attached with the SR name-label
<VM-NAME>-CONFIGDRIVE-ISO. While regular ISOs continue to attach with SR
name-label <VM-NAME>-ISO. VM which already have the configdrive ISO
attached before this fix will return an appropriate error and will need
to be stopped-start.
2024-06-27 16:10:33 +05:30
dahn
6b25ed7a02
prevent an NPE on an uninitialised TemplateObject (#8898)
* prevent an NPE on an uninitialised TemplateObject

* move npe handler up-stack

* Update engine/storage/image/src/main/java/org/apache/cloudstack/storage/image/store/TemplateObject.java

* catch yet one level up

* Update engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java

* Update engine/storage/image/src/main/java/org/apache/cloudstack/storage/image/store/TemplateObject.java

* extra guard

* Revert "prevent an NPE on an uninitialised TemplateObject"

This reverts commit e602a65ea62e4707828483a4ddea288d81ff06f5.
2024-06-26 21:02:08 +05:30