441 Commits

Author SHA1 Message Date
Wei Zhou
d7ca05e5ba
Build: drop EL7 support, support JRE17 for packages and sonar check (#8609)
This PR fixes the issue with sonar check

```
Error:  Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar (default-cli) on project cloudstack:
Error:
Error:  The version of Java (11.0.22) used to run this analysis is deprecated, and SonarCloud no longer supports it. Please upgrade to Java 17 or later.
Error:  You can find more information here: https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/
```

main changes
- Support build/packaging using JDK17
- Still supports JDK11 for building
- Support JRE17 for use in production installation
- Drop EL7 support

The community packages will be still packaged using JDK11. 
If uses want, they can build by JDK17 as well.

Signed-off-by: Wei Zhou <wei.zhou@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-08-27 16:09:10 +05:30
Nicolas Vazquez
296a44e932
Fix main branch issues (#9553) 2024-08-20 15:26:41 +02:00
Daan Hoogland
5bf81cf002 Merge release branch 4.19 to main
* 4.19:
  linstor: Improve copyPhysicalDisk performance (#9417)
2024-08-20 11:20:39 +02:00
Daan Hoogland
3d8d4875fe Merge release branch 4.18 to 4.19
* 4.18:
  linstor: Improve copyPhysicalDisk performance (#9417)
2024-08-20 11:19:44 +02:00
Rene Peinthor
27f23f4f75
linstor: Improve copyPhysicalDisk performance (#9417)
Tell qemu-img that we don't want to use a write cache (we are a block device)
and also specify that we have zeroed devices in most cases.
2024-08-20 11:17:57 +02:00
Suresh Kumar Anaparti
3faf7cd2f1
Updating pom.xml version numbers for release 4.19.2.0-SNAPSHOT
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2024-07-19 10:29:26 +05:30
Suresh Kumar Anaparti
9f4c895974
Updating pom.xml version numbers for release 4.19.1.0
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2024-07-15 17:19:29 +05:30
John Bampton
c923e673cf
pre-commit: add XML files to the trailing-whitespace check (#9131) 2024-07-12 09:42:54 +02:00
Abhishek Kumar
3c9bc07d06 Merge remote-tracking branch 'apache/4.19' 2024-07-06 13:01:50 +05:30
Vishesh
ba85b4d4ff
Merge release branch 4.18 to 4.19
* 4.18:
  linstor: deleteAsync fallback to volume UUID if path not set yet (#9325)
  linstor: do not use the same static ApiClient for all calls (#9326)
2024-07-05 12:45:27 +05:30
Rene Peinthor
eb1a0ef7f2
linstor: deleteAsync fallback to volume UUID if path not set yet (#9325) 2024-07-04 17:35:01 +05:30
Vishesh
7c32bd2506
Fixup main build errors (#9330)
* Fixup main build errors

* Fixup flaky test

* Address comments
2024-07-04 13:00:37 +05:30
Rene Peinthor
121a35d666
linstor: do not use the same static ApiClient for all calls (#9326)
The client.setBasePath() would overwrite the Linstor controller IP/host
for all current client users. This is basically a race condition
that triggered as soon as you had configured 2 different primary storages
with different Linstor controllers.
2024-07-03 16:13:16 +05:30
Vishesh
0ec7c72875
Merge branch '4.19' 2024-07-01 12:41:45 +05:30
Abhisar Sinha
063dc60114
Change storage pool scope from Cluster to Zone and vise versa (#8875)
* New feature: Change storage pool scope

* Added checks for Ceph/RBD

* Update op_host_capacity table on primary storage scope change

* Storage pool scope change integration test

* pull 8875 : Addressed review comments

* Pull 8875: remove storage checks, AbstractPrimayStorageLifeCycleImpl class

* Pull 8875: Fixed integration test failure

* Pull 8875: Review comments

* Pull 8875: review comments + broke changeStoragePoolScope into smaller functions

* Added UT for changeStoragePoolScope

* Rename AbstractPrimaryDataStoreLifeCycleImpl to BasePrimaryDataStoreLifeCycleImpl

* Pull 8875: Dao review comments

* Pull 8875: Rename changeStoragePoolScope.vue to ChangeStoragePoolScope.vue

* Pull 8875: Created a new smokes test file + A single warning msg in ui

* Pull 8875: Added cleanup in test_primary_storage_scope.py

* Pull 8875: Type in en.json

* Pull 8875: cleanup array in test_primary_storage_scope.py

* Pull:8875 Removing extra whitespace at eof of StorageManagerImplTest

* Pull 8875: Added UT for PrimaryDataStoreHelper and BasePrimaryDataStoreLifeCycleImpl

* Pull 8875: Added license header

* Pull 8875: Fixed sql query for vmstates

* Pull 8875: Changed icon plus info on disabled mode in apidoc

* Pull 8875: Change scope should not work for local storage

* Pull 8875: Change scope completion event

* Pull 8875: Added api findAffectedVmsForStorageScopeChange

* Pull 8875: Added UT for findAffectedVmsForStorageScopeChange and removed listByPoolIdVMStatesNotInCluster

* Pull 8875: Review comments + Vm name in response

* Pull 8875: listByVmsNotInClusterUsingPool was returning duplicate VM entries because of multiple volumes in the VM satisfying the criteria

* Pull 8875: fixed listAffectedVmsForStorageScopeChange UT

* listAffectedVmsForStorageScopeChange should work if the pool is not disabled

* Fix listAffectedVmsForStorageScopeChangeTest UT

* Pull 8875: add volume.removed not null check in VmsNotInClusterUsingPool query

* Pull 8875: minor refactoring in changeStoragePoolScopeToCluster

* Update server/src/main/java/com/cloud/storage/StorageManagerImpl.java

* fix eof

* changeStoragePoolScopeToZone should connect pool to all Up hosts

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-29 10:03:34 +05:30
Suresh Kumar Anaparti
2ca1b474bd
PowerFlex/ScaleIO SDC client connection improvements (#9268)
* Mitigation for non-scalable Powerflex/ScaleIO clients
- Added ScaleIOSDCManager to manage SDC connections, checks clients limit, prepare and unprepare SDC on the hosts.
- Added commands for prepare and unprepare storage clients to prepare/start and stop SDC service respectively on the hosts.
- Introduced config 'storage.pool.connected.clients.limit' at storage level for client limits, currently support for Powerflex only.

* tests issue fixed

* refactor / improvements

* lock with powerflex systemid while checking connections limit

* updated powerflex systemid lock to hold till sdc preparation

* Added custom stats support for storage pool, through listStoragePools API

* code improvements, and unit tests

* unit tests fixes

* Update config 'storage.pool.connected.clients.limit' to dynamic, and some improvements

* Stop SDC on host after migration if no volumes mapped to host

* Wait for SDC to connect after scini service start, and some log improvements

* Do not throw exception (log it) when SDC is not connected while revoking access for the powerflex volume

* some log improvements
2024-06-29 10:01:50 +05:30
Vishesh
90fe1d5fdc
Merge branch '4.19' 2024-06-29 03:35:24 +05:30
Vishesh
bcbf152a05
Merge branch '4.18' into 4.19 2024-06-28 20:14:21 +05:30
Suresh Kumar Anaparti
983f164c57
Fixed src datastore on copy check for PowerFlex/ScaleIO storage driver (#9310) 2024-06-28 18:46:06 +05:30
Abhisar Sinha
644f3a3f48
Add, Delete Storage Pool commands should be able execute on a host in maintenance (#9301)
* Restart agent when host comes out of maintenance

* Don't send CreateStoragePoolCommand to hosts in maintenance mode

* CreateStoragePoolCommand can run when host in maintenance. Reverted the change to restart agent when host was already up and in maintenance

* Reverted changes done to ResourceManagerImplTest
2024-06-28 18:18:08 +05:30
slavkap
6c06e85c80
Temporarily backup StorPool volume before expunge (#8843)
* Temporarily backup StorPool volume before expunge

Sometimes the users delete the volumes by mistake. This enhancment
provides a solution to backup the volume before it's deleted. The user
will be able to see the snapshot in CloudStack UI/CLI and create only a
volume from it.
A task will check (by default on every 5mins) if the snapshots are
deleted from StorPool

Global settings to enable the delay delete option:

`storpool.delete.after.interval` - The interval (in seconds) after the StorPool snapshot will be deleted
`storpool.list.snapshots.delete.after.interval` - The interval (in seconds) to fetch the StorPool snapshots with deleteAfter flag

Minor fix when deleting snapshots

* added Apache licence

* addressed comments
2024-06-26 13:58:04 +05:30
Abhisar Sinha
4eb43651e2
Ability to specify NFS mount options while adding a primary storage and modify them on a pre-existing primary storage (#8947)
* Ability to specify NFS mount options while adding a primary storage and modify it later

* Pull 8947: Rename all occurrence of nfsopt to nfsMountOpt and added nfsMountOpts to ApiConstants

* Pull 8947: Refactor code - move into separate methods

* Pull 8947: CollectionsUtils.isNotEmpty and switch statement in LibvirtStoragePoolDef.java

* Pull 8947: UI - cancel maintainenace will remount the storage pool and apply the options

* Pull 8947: UI - moved edit NFS mount options to edit Primary Storage form

* Pull 8947: UI - moved 'NFS Mount Options' to below 'Type' in dataview

* Pull 8947: Fixed message in AddPrimaryStorage.vue

* Pull 8947: Convert _nfsmountOpts to Set in libvirtStoragePoolDef

* Pull 8947: Throw exception and log error if mount fails due to incorrect mount option

* Pull 8947: Added UT and moved integration test to component/maint

* Pull 8947: Review comments

* Pull 8947: Removed password from integration test

* Pull 8947: move details allocation to inside the if loop in getStoragePoolNFSMountOpts

* Pull 8947: Fixed a bug in AddPrimaryStorage.vue

* Pull 8947: Pool should remain in maintenance mode if mount fails

* Pull 8947: Removed password from integration test

* Pull 8947: Added UT

* Pull 8875: Fixed a bug in CloudStackPrimaryDataStoreLifeCycleImplTest

* Pull 8875: Fixed a bug in LibvirtStoragePoolDefTest

* Pull 8947: minor code restructuring

* Pull 8947 : added some ut for coverage

* Fix LibvirtStorageAdapterTest UT
2024-06-25 23:45:35 +05:30
Vishesh
3923f80c22
Merge branch '4.19' 2024-06-25 18:53:57 +05:30
Rene Glover
6ee6603359
Updates to HPE-Primera and Pure FlashArray Drivers to use Host-based VLUN Assignments (#8889)
* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* improve error handling when copying volumes to add precision to which step failed

* rename pure volume before delete to avoid conflicts if the same name is used before its expunged on the array

* remove dead code in AdaptiveDataStoreLifeCycleImpl.java

* Fix issues found in PR checks

* fix session refresh TTL logic

* updates from PR comments

* logic to delete by path ONLY on supported OUI

* fix to StorageSystemDataMotionStrategy compile error

* change noisy debug message to trace message

* fix double callback call in handleVolumeMigrationFromNonManagedStorageToManagedStorage

* fix for flash array delete error

* fix typo in StorageSystemDataMotionStrategy

* change copyVolume to use writeback to speed up copy ops

* remove returning PrimaryStorageDownloadAnswer when connectPhysicalDisk returns false during KVMStorageProcessor template copy

* remove change to only set UUID on snapshot if it is a vmSnapshot

* reverting change to UserVmManagerImpl.configureCustomRootDiskSize

* add error checking/simplification per comments from @slavkap

* Update engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* address PR comments from @sureshanaparti

---------

Co-authored-by: GLOVER RENE <rg9975@cs419-mgmtserver.rg9975nprd.app.ecp.att.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-25 10:35:39 +05:30
slavkap
8b07b66f14
Fix volume snapshot of encrypted NFS/StorPool volume (#8873)
* Fix volume snapshot of encrypted NFS/StorPool volume

* remove comments

* removed invoking the real qemu convert command

* fix UnsatisfiedLink error in unit tests

* addressed comments

extracted method
2024-06-24 13:09:21 +05:30
Daan Hoogland
8806e44e4f Merge release branch 4.19 to main
* 4.19:
  [linstor] Fix revertSnapshot (#9271)
2024-06-20 11:19:28 +02:00
Michael Wodniok
7dce3d87d4
[linstor] Fix revertSnapshot (#9271)
Signed-off-by: Michael Wodniok (WorNet AG) <michael.wodniok@wor.net>
Co-authored-by: Michael Wodniok <michael.wodniok@wor.net>
2024-06-20 10:52:49 +02:00
Wei Zhou
3a5c4a0cba
fix build error with LinstorVMSnapshotStrategy 2024-06-14 23:03:03 +02:00
Daan Hoogland
cb9b3134f7 Merge branch '4.19' 2024-06-14 10:30:10 +02:00
Rene Peinthor
37f4398c80
linstor: Support VM-Instance Disk snapshots (#8796)
* linstor: update to java-linstor 0.5.1

* linstor: Support VM-Instance Disk snapshots

This adds VM-Instance disk snapshot support for
Linstor primary storage. Instance snapshots are stored on
the used Linstor storage pool backend and can be converted
into regular volume snapshots and also reverted.

Instance VM snapshots are not fully atomic but with the
create multi snapshot feature as good as it gets.
Snapshots are done over multiple volumes in the same devicemanager run.
2024-06-13 15:26:33 +05:30
Vishesh
6fe835e118
Merge branch '4.19' 2024-06-13 11:57:55 +05:30
Suresh Kumar Anaparti
4ec0f823cf
ScaleIO volume live migration - use usable bytes from source disk to format the destination disk (#9174) 2024-06-12 13:58:10 +05:30
Daan Hoogland
b29391c329 Merge LTS branch '4.19' into main 2024-06-07 09:53:57 +02:00
Daan Hoogland
c779b1c616 Merge branch '4.18' into 4.19 2024-06-06 11:24:09 +02:00
Rene Peinthor
2339412f73
linstor: make getDevicePath more robust (#9143) 2024-06-06 09:49:03 +02:00
John Bampton
28e8e2d009
pre-commit: add hook to trim trailing whitespace (#8205) 2024-05-28 09:01:30 +02:00
Vishesh
21af134087
Fix exceeding of resource limits with powerflex (#9008)
* Fix exceeding of resource limits with powerflex

* Add e2e tests

* Update server/src/main/java/com/cloud/vm/UserVmManagerImpl.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* fixup

---------

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-05-08 20:55:19 +05:30
Vishesh
e9ff2707bb
Merge branch '4.19' 2024-05-07 12:47:48 +05:30
Rene Peinthor
f80d205284
linstor: Fix volume format and make resource available on copy target (#8811)
Linstor primary storage forgot to make sure
the volume download/copy target has a Linstor resource available.
2024-05-06 11:00:22 +02:00
Daan Hoogland
0d8f7d4003 Merge release branch 4.19 to main
* 4.19:
  linstor: disconnect-disk also search for resource name in Linstor (#9035)
  ui: add support to change Account role for admins (#9012)
  Use parameter dcId as wrapper to prevent NPE (#8986)
2024-05-06 10:36:06 +02:00
Daan Hoogland
92ba476593 Merge release branch 4.18 to 4.19
* 4.18:
  linstor: disconnect-disk also search for resource name in Linstor (#9035)
2024-05-06 10:35:27 +02:00
Rene Peinthor
ea11128cb3
linstor: disconnect-disk also search for resource name in Linstor (#9035)
disconnectPhysicalDisk(String, KVMStoragePool) seems to calls the plugin
with the resource name instead of the device path, so we also have
to search for resource names, while cleaning up.
2024-05-06 09:05:31 +02:00
Daan Hoogland
e61f3bae4d Merge branch '4.19' 2024-04-29 11:37:40 +02:00
Rohit Yadav
3de1f8b4ba Merge remote-tracking branch 'origin/4.18' into 4.19
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-04-29 13:44:34 +05:30
Rene Peinthor
9d5d4e5564
linstor: cleanup diskless nodes on disconnect (#8790) 2024-04-26 14:25:07 +02:00
Daan Hoogland
a358c9a410 Merge release branch 4.19 to main
* 4.19:
  New feature: Import/Unamange DATA volume from storage pool (#8808)
2024-04-23 16:13:16 +02:00
Wei Zhou
0b857def68
New feature: Import/Unamange DATA volume from storage pool (#8808) 2024-04-23 16:05:59 +02:00
Daan Hoogland
66c10bf42f logger issues 2024-04-23 12:55:17 +02:00
Rohit Yadav
93d4816ee8 Merge remote-tracking branch 'origin/4.19'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

 Conflicts:
	plugins/storage/volume/linstor/src/main/java/com/cloud/hypervisor/kvm/storage/LinstorStorageAdaptor.java
2024-04-23 15:24:19 +05:30
Rohit Yadav
0fa71f5696 Merge remote-tracking branch 'origin/4.18' into 4.19 2024-04-23 15:21:44 +05:30