2951 Commits

Author SHA1 Message Date
Vishesh
90fe1d5fdc
Merge branch '4.19' 2024-06-29 03:35:24 +05:30
Suresh Kumar Anaparti
46f672563e
Improve migration of external VMware VMs into KVM cluster (#8815)
* Create/Export OVA file of the VM on external vCenter host, to temporary conversion location (NFS)

* Fixed ova issue on untar/extract ovf from ova file
"tar -xf" cmd on ova fails with "ovf: Not found in archive" while extracting ovf file

* Updated VMware to KVM instance migration using OVA

* Refactoring and cleanup

* test fixes

* Consider zone wide pools in the destination cluster for instance conversion

* Remove local storage pool support as temporary conversion location
- OVA export not possible as the pool is not accessible outside host, NFS pools are supported.

* cleanup unused code

* some improvements, and refactoring

* import nic unit tests

* vmware guru unit tests

* Separate clone VM and create template file for VMware migration
- Export OVA (of the cloned VM) to the conversion location takes time.
- Do any validations with cloned VM before creating the template (and fail early).
- Updated unit tests.

* Check conversion support on host before clone vm / create template on vmware (and fail early)

* minor code improvements

* Auto select the host with instance conversion capability

* Skip instance conversion supported response param for non-KVM hosts

* Show supported conversion hosts in the UI

* Skip persistence map update if network doesn't exist

* Added support to export OVA from KVM host, through ovftool (when installed in KVM host)

* Updated importvm api param 'usemsforovaexport' to 'forcemstodownloadvmfiles', to be generic

* Updated hardcoded UI messages with message labels

* Updated UI to support importvm api param - forcemstodownloadvmfiles

* Improved instance conversion support checks on ubuntu hosts, and for windows guest vms

* Use OVF template (VM disks and spec files) for instance conversion from VMware, instead of OVA file
 - this would further increase the migration performance (as it reduces the time for OVA preparation / archiving of the VM files into a single file)

* OVF export tool parallel threads code improvements

* Updated 'convert.vmware.instance.to.kvm.timeout' config default value to 3 hrs

* Config values check & code improvements

* Updated import log, with time taken and vm details

* Support for parallel downloads of VMware VM disk files while exporting OVF from MS, and other changes below.
- Skip clone for powered off VMs
- Fixes to support standalone host (with its default datacenter)
- Some code improvements

* rebase fixes

* rebase fixes

* minor improvement

* code improvements - threads configuration, and api parameter changes to import vm files

* typo fix in error msg
2024-06-27 21:14:13 +05:30
Abhishek Kumar
53faf0f66a
xenserver: attach regular iso with configdrive (#9216)
* xenserver: attach regular iso with configdrive

Fixes #7902

This PR allows attaching a regular ISO to a VM when it already has the
config drive ISO attached.
Config-drive ISO is now attached with the SR name-label
<VM-NAME>-CONFIGDRIVE-ISO. While regular ISOs continue to attach with SR
name-label <VM-NAME>-ISO. VM which already have the configdrive ISO
attached before this fix will return an appropriate error and will need
to be stopped-start.
2024-06-27 16:10:33 +05:30
Wei Zhou
22cd00ffb1
veeam: fix issues with PreSetup and DVS and Solidfire (#9256)
* Veeam: find storage pool by path for PreSetup and VMFS

* Veeam: support VMware distributed virtual switch

* Veeam: sync volumes on Solidfire after backup restoration

user faced the issue that backup is restored but the DATA disk is gone (ROOT disk is ok)
```
2024-05-03 12:00:32,868 ERROR [o.a.c.b.BackupManagerImpl] (API-Job-Executor-13:ctx-aa8a1d85 job-149661 ctx-73328567) (logid:6510cf06) Failed to import VM [vmInternalName: i-169-9679-VM] from backup restoration [{"backupType":"Full","externalId":"821ca400-a5da-4282-bf3f-7c7e38a6cdb4","id":257,"uuid":"69399101-5cbd-461c-8a48-f0c70eac0b24","vmId":9679}] with hypervisor [type: VMware] due to: [Couldn't find storage pool -iqn.2010-01.com.solidfire:3p53.data-9679.221-0].
```

On managed storage, the datastore name of DATA disk is determined by the iscsi_name of the volume.

* Veeam: set correct path for DATA disks on solidfire
2024-06-26 18:02:25 +05:30
Wei Zhou
3a0c312591
fix build error with FiberChannelAdapter after forward-merge 2024-06-26 08:35:13 +02:00
Abhisar Sinha
4eb43651e2
Ability to specify NFS mount options while adding a primary storage and modify them on a pre-existing primary storage (#8947)
* Ability to specify NFS mount options while adding a primary storage and modify it later

* Pull 8947: Rename all occurrence of nfsopt to nfsMountOpt and added nfsMountOpts to ApiConstants

* Pull 8947: Refactor code - move into separate methods

* Pull 8947: CollectionsUtils.isNotEmpty and switch statement in LibvirtStoragePoolDef.java

* Pull 8947: UI - cancel maintainenace will remount the storage pool and apply the options

* Pull 8947: UI - moved edit NFS mount options to edit Primary Storage form

* Pull 8947: UI - moved 'NFS Mount Options' to below 'Type' in dataview

* Pull 8947: Fixed message in AddPrimaryStorage.vue

* Pull 8947: Convert _nfsmountOpts to Set in libvirtStoragePoolDef

* Pull 8947: Throw exception and log error if mount fails due to incorrect mount option

* Pull 8947: Added UT and moved integration test to component/maint

* Pull 8947: Review comments

* Pull 8947: Removed password from integration test

* Pull 8947: move details allocation to inside the if loop in getStoragePoolNFSMountOpts

* Pull 8947: Fixed a bug in AddPrimaryStorage.vue

* Pull 8947: Pool should remain in maintenance mode if mount fails

* Pull 8947: Removed password from integration test

* Pull 8947: Added UT

* Pull 8875: Fixed a bug in CloudStackPrimaryDataStoreLifeCycleImplTest

* Pull 8875: Fixed a bug in LibvirtStoragePoolDefTest

* Pull 8947: minor code restructuring

* Pull 8947 : added some ut for coverage

* Fix LibvirtStorageAdapterTest UT
2024-06-25 23:45:35 +05:30
Vishesh
3923f80c22
Merge branch '4.19' 2024-06-25 18:53:57 +05:30
Suresh Kumar Anaparti
620ed164d8
VMware: Improve error messaging / logs when starting non-user VMs, and secondary storage not available or doesn't have enough capacity (#9207) 2024-06-25 12:25:42 +05:30
Rene Glover
6ee6603359
Updates to HPE-Primera and Pure FlashArray Drivers to use Host-based VLUN Assignments (#8889)
* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* improve error handling when copying volumes to add precision to which step failed

* rename pure volume before delete to avoid conflicts if the same name is used before its expunged on the array

* remove dead code in AdaptiveDataStoreLifeCycleImpl.java

* Fix issues found in PR checks

* fix session refresh TTL logic

* updates from PR comments

* logic to delete by path ONLY on supported OUI

* fix to StorageSystemDataMotionStrategy compile error

* change noisy debug message to trace message

* fix double callback call in handleVolumeMigrationFromNonManagedStorageToManagedStorage

* fix for flash array delete error

* fix typo in StorageSystemDataMotionStrategy

* change copyVolume to use writeback to speed up copy ops

* remove returning PrimaryStorageDownloadAnswer when connectPhysicalDisk returns false during KVMStorageProcessor template copy

* remove change to only set UUID on snapshot if it is a vmSnapshot

* reverting change to UserVmManagerImpl.configureCustomRootDiskSize

* add error checking/simplification per comments from @slavkap

* Update engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* address PR comments from @sureshanaparti

---------

Co-authored-by: GLOVER RENE <rg9975@cs419-mgmtserver.rg9975nprd.app.ecp.att.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-25 10:35:39 +05:30
slavkap
8b07b66f14
Fix volume snapshot of encrypted NFS/StorPool volume (#8873)
* Fix volume snapshot of encrypted NFS/StorPool volume

* remove comments

* removed invoking the real qemu convert command

* fix UnsatisfiedLink error in unit tests

* addressed comments

extracted method
2024-06-24 13:09:21 +05:30
Suresh Kumar Anaparti
c17aa0d9ad
Import Remote KVM VM logging improvements (#9284) 2024-06-24 11:34:37 +05:30
Rene Peinthor
f4612c51ec
libvirtstorage: Make sure netfs storage was really mounted (#8887) 2024-06-23 19:41:02 +05:30
Vishesh
674495b162
Fixup startVM on simulator (#9199) 2024-06-21 15:53:45 +05:30
Suresh Kumar Anaparti
5ab23cd9c9
Timeout config to copy the disks of remote KVM instance while importing the instance from an external host (#9213)
* Added timeout config to copy the disks of remote KVM instance while importing the instance from an external host

* Updated copy config units to mins

* Cleanup remote converted file and local file when copy failed
2024-06-21 10:28:18 +05:30
Wei Zhou
fbeee49137
fix build errors after forward-merge 2024-06-20 09:55:35 +02:00
Wei Zhou
d81ffd2d86
Merge remote-tracking branch 'apache/4.19' 2024-06-20 08:57:17 +02:00
Daan Hoogland
3997e59678 Merge release branch 4.18 to 4.19
* 4.18:
  Update extraconfig for platform param in xen/xcpng (#9248)
2024-06-19 18:55:29 +02:00
Suresh Kumar Anaparti
cc52b38e54
Update extraconfig for platform param in xen/xcpng (#9248)
* Update extraconfig for platform param in xen/xcpng

* Fix map param key, not to replace '-' with '_' (replace only applicable to param / map-param)

* Added unit tests

* Add license for tests file
2024-06-18 23:39:50 +05:30
Daan Hoogland
373f017002 Merge branch '4.19' 2024-06-18 19:58:43 +02:00
Wei Zhou
f360f7048d
vmware: do not tear down vm disks if deploy-as-is vm has vm snapshots (#9243) 2024-06-18 08:28:20 +02:00
Daan Hoogland
cb9b3134f7 Merge branch '4.19' 2024-06-14 10:30:10 +02:00
Vishesh
74f5e52e6e
Fix unit test failure (#9238) 2024-06-13 16:06:35 +05:30
Wei Zhou
cb48202b34
fix build error LibvirtGetVolumeStatCommandWrapper 2024-06-13 10:43:35 +02:00
Vishesh
6fe835e118
Merge branch '4.19' 2024-06-13 11:57:55 +05:30
Wei Zhou
b2ef53b8a2
kvm: replace ISO path in vm XML configuration during vm migration (#9212)
* kvm: replace ISO path in vm XML configuration during vm migration

* Update 9212: address comments

* kvm: fix vm migration if there are multiple image stores
2024-06-12 16:01:23 +02:00
Suresh Kumar Anaparti
4ec0f823cf
ScaleIO volume live migration - use usable bytes from source disk to format the destination disk (#9174) 2024-06-12 13:58:10 +05:30
Suresh Kumar Anaparti
2e3f76ec03
Improve error messaging / logs when listing VMs on the remote KVM host (for import) (#9204) 2024-06-11 14:48:21 +02:00
Harikrishna
acae5c5b9e
kvm: Update the java doc for the method disconnectPhysicalDiskByPath (#9210)
This PR addresses the issue #8789

The original issue is disconnectPhysicalDiskByPath() implementation in FibreChannelAdaptor always returns true irrespective of the success of the operation. This was already fixed in the PR #8889 .

Ideally this method has to be called after choosing the right adapter based on the storage pool type of the volume path, but currently it is just called in a loop.
05b9b6e2e7/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java (L200-L212)

while trying to fix the case of running into the loop of all adapters by somehow passing the storage pool type to that caller cleanup() method but this is touching all over the code (which I fear it creates other regressions), instead I feel we can keep it the current way only since Fibrechannel adapter has already fixed.

In this PR I've added the java doc explaining the method and situation.
2024-06-11 14:44:46 +05:30
Rohit Yadav
8d211e75c0 Merge remote-tracking branch 'origin/4.19' 2024-06-10 16:18:41 +05:30
Abhishek Kumar
10f4de0318
kvm: consider provisioning type for local data volumes (#9141)
Fixes #8644

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-10 11:38:31 +03:00
dahn
0c615312f9
log messages (#9093)
* log messages

* Apply suggestions from code review

Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>

* Update plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java

Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>

---------

Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2024-06-06 12:41:11 +05:30
Wei Zhou
5433e775e5
New feature: Implicit host tags (#8929)
* Merge two HostTagVO and HostTagDaoImpl

* Implicit host tags

* PR8929: add since

* Update variable names

* Update 8929: add unit test in LibvirtComputingResourceTest

* Update 8929: add explicithosttags in response

* Update 8929 UI: Update explicit host tags

* Update 8929: remove host tags and change labels on UI

* Update 8929: update host_view to use explicit_host_tags.is_tag_a_rule

* Update: ui polish for host tags

* Update 8929: fix UI error if no host tags
2024-05-30 17:21:13 +05:30
John Bampton
28e8e2d009
pre-commit: add hook to trim trailing whitespace (#8205) 2024-05-28 09:01:30 +02:00
Wei Zhou
f41c934c94
Fix build error with MigrateKVMAsync 2024-04-29 12:33:23 +02:00
Daan Hoogland
e61f3bae4d Merge branch '4.19' 2024-04-29 11:37:40 +02:00
Rohit Yadav
3de1f8b4ba Merge remote-tracking branch 'origin/4.18' into 4.19
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-04-29 13:44:34 +05:30
João Jandre
cec6ade257
change live migration API used on kvm (#8952) 2024-04-25 09:35:25 +02:00
Marcus Sorensen
27124c1031
Add ability to set cpu.threadspercore similar to existing cpu.corespersocket (#8850)
* Add ability to set cpu.threadspercore similar to existing cpu.corespersocket

* add cpu.threadspercore to VM and template detail options

* Update plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* add vm detail for KVM

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-04-24 18:01:21 +05:30
Daan Hoogland
a358c9a410 Merge release branch 4.19 to main
* 4.19:
  New feature: Import/Unamange DATA volume from storage pool (#8808)
2024-04-23 16:13:16 +02:00
Wei Zhou
0b857def68
New feature: Import/Unamange DATA volume from storage pool (#8808) 2024-04-23 16:05:59 +02:00
Rohit Yadav
93d4816ee8 Merge remote-tracking branch 'origin/4.19'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

 Conflicts:
	plugins/storage/volume/linstor/src/main/java/com/cloud/hypervisor/kvm/storage/LinstorStorageAdaptor.java
2024-04-23 15:24:19 +05:30
Rohit Yadav
0fa71f5696 Merge remote-tracking branch 'origin/4.18' into 4.19 2024-04-23 15:21:44 +05:30
Rohit Yadav
5a52ca78ae
kvm: export sysinfo for arm64 domains for cloud-init to work (#8940)
This fixes a limitation for arm64/aarch64 KVM hosts to correctly export
the product name via sysconfig attribute. Without this `cloud-init`
doesn't function correctly on arm64 platforms.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-04-19 21:23:49 +02:00
Daan Hoogland
3cabe69cd5 Merge release branch 4.19 to main
* 4.19:
  protect against null-path (#8915)
  UI: Fix missing locale strings for Status widget (#8792)
  Add a shutdownhook to remove jobs owned by the process (#8896)
2024-04-19 12:45:30 +02:00
Daan Hoogland
78e07cff62 Merge release branch 4.18 to 4.19
* 4.18:
  protect against null-path (#8915)
  UI: Fix missing locale strings for Status widget (#8792)
  Add a shutdownhook to remove jobs owned by the process (#8896)
2024-04-19 12:43:34 +02:00
dahn
7affbb1dac
protect against null-path (#8915)
Co-authored-by: Vladimir Dombrovski <vladimir.dombrovski@bso.co>
Co-authored-by: Vishesh <vishesh92@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-04-19 12:23:31 +02:00
Suresh Kumar Anaparti
dfd5158d67
Allow kvm storage plugin to customize diskdef, add geometry (#8839)
* Allow kvm storage plugin to customize diskdef, add geometry

* formatting update

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
2024-04-18 18:21:17 +05:30
Wei Zhou
38ca11f99a
build: fix build failure after merge-forward 2024-04-17 16:40:31 +02:00
João Jandre
8a101fbbc1 Updating pom.xml version numbers for release 4.18.3.0-SNAPSHOT
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-04-17 11:11:57 -03:00
GaOrtiga
31f28f6516
Limit infrastructure details revealed to the end user after a failed migration on KVM (#8680)
* remove details when error occurs

* Update plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

---------

Co-authored-by: Gabriel <gabriel.fernandes@scclouds.com.br>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2024-04-16 09:58:35 -03:00