35999 Commits

Author SHA1 Message Date
Rohit Yadav
18e7276df0 storage: allow VM snapshots without memory for KVM when global setting allows (#8062)
This removes the conditional logic where comment notest to remove it
after PR #5297 is merged that is applicable for ACS 4.18+. Only when the
global setting is enabled and memory isn't selected, VM snapshot could
be allowed for VMs on KVM that have qemu-guest-agent running.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 8350ce5aa40241d683749c1257813fa649e998be)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-11 21:01:31 +05:30
Nicolas Vazquez
fe3cf81739 Fix #303: Run recalculation recurrent task only in the longest running management server (#335)
* Run recalculation recurrent task only in the longest running management server

* Fix timeout and recalculation when no child domains

(cherry picked from commit 3f35e3a6ef24017d2f63613ca4362521b1ee78b6)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-11 20:52:57 +05:30
Rohit Yadav
a54f37890c
Merge pull request #322 from shapeblue/4.18-migration-issue320
Issue-320: Backport and changes to help with 4.18 migration
2023-10-10 13:15:33 +05:30
dahn
81f9da037b ldap trust map cleanup on domain delete (#7915)
Co-authored-by: Wei Zhou <weizhou@apache.org>
(cherry picked from commit 09ae0499b21d680425db1738f60ba12b5e16f3bc)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-09 19:15:47 +05:30
Pearl Dsilva
73c86b8a30 VR live patching: Allow live patch of VPC VRs even if networks are in allocated / shutdown state (#7958)
(cherry picked from commit 951ba04cf0eae3f48207306f0035a2292b55506d)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-09 19:15:32 +05:30
Fabricio Duarte
4f86c452de Fix 2FA becoming enabled when the user inputs the wrong code during setup (#7972)
(cherry picked from commit ef742210b5d05c064253eb6b0bcf76d282572a33)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-09 19:15:25 +05:30
Peinthor Rene
1fdaec5c90 linstor: resize root disk on offerings with different size (#7952)
(cherry picked from commit 96205a51ef918313ad8b3288758e15184a710b91)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-09 19:14:28 +05:30
Wei Zhou
24346b5d65 server: fix mysql error when list Shared templates for project (#8020)
(cherry picked from commit 3d8cc63bc32ee374d22e42de44619bdce65c5e33)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-09 19:14:20 +05:30
Wei Zhou
4498c7807b systemvm: check if patchfile exists and is valid tarball before patching (#7953)
(cherry picked from commit 51add0a066207a4f926e8fd808b4913078f3d8a5)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-09 19:14:14 +05:30
Daniel Augusto Veronezi Salvador
5b7837eae1 Fix: Convert volume to another directory instead of copying it while taking volume snapshots on KVM (#8041)
(cherry picked from commit 9b8eaeea78a203926f0fc00e948ce30cfb2eb519)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-09 19:12:40 +05:30
Oscar Sandoval
832995abd6 Improve template download error message (#8029)
(cherry picked from commit c0128e2f9a4cade8b9129de99916970c72bc212c)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-09 19:12:12 +05:30
Wei Zhou
1b36cd9210 test: fix test_vm_autoscaling.py which does not work due to userdata improvement (#7921)
(cherry picked from commit 78411fd405c6dfe83381f010c143d6bb90b6b7e4)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-29 14:32:29 +05:30
Rohit Yadav
1b35a5aa1c Fix build, missing imports
Note: these are however redundant, both on main branch and the private
branch across two classes.

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-28 14:31:24 +05:30
Abhishek Kumar
160a13a029 userdata: fix append scenarios (#7741)
Fixes case of appending userdata when both template and vm data are either shellscript or cloudconfig
Fixes error when appending gzip userdata
Fixes case when userdata manual text from VM is not getting decoded-encoded correctly.
Fixes case of appending multipart data when both template and vm data contain same format types.
Refactor - moved validateUserData method to UserDataManager class
Refactor userdata test to check resultant multipart userdata thoroughly

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
(cherry picked from commit 729e6d144655bd26e6453dcc01a7e6f0d5c8f50e)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-28 12:48:46 +05:30
Marcus Sorensen
c01ed90569 Trigger out of band VM state update via libvirt event when VM stops (#7963)
* Trigger out of band VM state update via libvirt event when VM stops

* Add License headers, refactor nested try

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
(cherry picked from commit 3694667f5074018d6df58fac5e6a2803204f295b)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-28 12:22:15 +05:30
Rohit Yadav
e77107819b kvm: skip KVMHostInfoTest when build OS isn't Linux
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 17:47:10 +05:30
Marcus Sorensen
33a5c159a3 KVM Agent config to reserve dom0 CPUs (#326)
Cherry-picked from 1a722edfce6234319c5eaecf3cd0076fa9690267

Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-09-27 17:15:40 +05:30
Marcus Sorensen
d083df504d LibvirtServerDiscoverer should only process added KVM hosts (#324)
Don't use LibvirtServerDiscoverer's processHostAdded() in CustomServerDiscoverer

(cherry picked from commit ba0dfea6fd469063d6a154d2056f250feed84f10)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 17:07:16 +05:30
Marcus Sorensen
7ed872c1d6 Use direct download timeout configs for URL check (#316)
(cherry picked from commit 289fe4bd5e8fad79fc416a26c3345ff93efb466a)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 17:06:20 +05:30
Nicolas Vazquez
62fce86234 Generate cloud-init multipart user data for template append policy (#7643)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
(cherry picked from commit b1fc2798726bd5ca0782bce59749e8ee30399916)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 16:50:41 +05:30
Nicolas Vazquez
c6c32776b5 UI: Fix user role login due to missing API access on custom hypervisor name (#7939)
* UI: Fix user role login due to missing API access on custom hypervisor name

* Refactor to include the custom HW display name as part of the response of listCapabilities API

* Add since parameter

(cherry picked from commit 940733cedfb1b46930d280c76a5f436d9c01f553)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 16:19:51 +05:30
Nicolas Vazquez
4dfa38aae9 plugins: Add Custom hypervisor minimal changes (#7692)
Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor

This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack

The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.

CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately

This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)

Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin

- The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
- A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
   - In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
- The new Custom hypervisor type supports:
   - Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
   - Local storage (use hypervisor hosts local storage as primary storage)
   - Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
- The UI is also extended to display the new hypervisor type and the supported features listed above.
- The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.

This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.

- CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
- CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts

(cherry picked from commit 8b5ba13b817e7d53beb0e0012c27d065e96a708c)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 16:19:42 +05:30
Marcus Sorensen
fe50018fba Allow configkey to set 'cloud-name' cloud-init metadata (#7964)
* Allow configkey to set 'cloud-name' cloud-init metadata

* Update engine/api/src/main/java/com/cloud/vm/VirtualMachineManager.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update server/src/main/java/com/cloud/network/NetworkModelImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update server/src/main/java/com/cloud/network/router/CommandSetupHelper.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Revert "Update server/src/main/java/com/cloud/network/router/CommandSetupHelper.java"

This reverts commit 8abc3e38c4f973227e77253a8bcf3c94827d187f.

* Revert "Update server/src/main/java/com/cloud/network/NetworkModelImpl.java"

This reverts commit 7f239be919112913d764b141e943acca4473f755.

* Rework/Fix review code suggestions

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
(cherry picked from commit 155a30748cfdcdffe31ce3b13f54008323f4774a)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 15:47:58 +05:30
Oscar Sandoval
be2bea64a9 fix units label (#325)
(cherry picked from commit d708603b6346033f0d023edfebe85113fcc928eb)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 15:45:45 +05:30
Marcus Sorensen
a571cde7a9 Increase reserve on ScaleIO disk formatting for fragmentation (#317)
Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
(cherry picked from commit 59217e3fe19fc01863d678a34a618914afaf901b)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 15:43:14 +05:30
Marcus Sorensen
d8db09ac58 Publish event for VM.STOP when out of band stop is detected
Signed-off-by: Marcus Sorensen <mls@apple.com>
(cherry picked from commit 72a3d54db0a0bb0afbcbee35abc1e98ef102aa49)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 15:41:47 +05:30
Abhishek Kumar
0d9aa70edd test: enable and fix scaleio unit tests (#289)
Fixes #275

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
(cherry picked from commit c82bc16a20527d5d07c323d5323bccd56fc604e0)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 15:32:07 +05:30
Marcus Sorensen
5604638b84 Apple base416 passphrase enc (#240)
* Move PassphraseVO to use String instead of byte[] to support Encrypt annotation

* Check for unencrypted passphrases before migrating passphrase table

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>

Fixes #239

This PR moves PassphraseVO passphrase to String type. Since the
GenericDaoBase manipulates encrypted fields as Strings we don't improve
anything by handling as byte arrays. We still use byte arrays to pass
these values down to the agents and we can get some security gains
there.

This PR also handles cases where the passphrase field may be previously
unencrypted, and upgrades them to encrypted fields using the old
encryption during cloudstack-migrate-databases. Then the process can
upgrade to new encryption normally.
2023-09-27 13:24:13 +05:30
Marcus Sorensen
a9fb36174d Skip volume resize during service offering change when no size exists (#200) 2023-09-27 13:24:08 +05:30
Nicolas Vazquez
20952b4842 Auto Enable/Disable KVM hosts (#7170)
* Auto Enable Disable KVM hosts

* Improve health check result

* Fix corner cases

* Script path refactor

* Fix sonar cloud reports

* Fix last code smells

* Add marvin tests

* Fix new line on agent.properties to prevent host add failures

* Send alert on auto-enable-disable and add annotations when the setting is enabled

* Address reviews

* Add a reason for enabling or disabling a host when the automatic feature is enabled

* Fix comment on the marvin test description

* Fix for disabling the feature if the admin has manually updated the host resource state before any health check result

(cherry picked from commit be66eb2a35bd6a5aae74f97a9140a9ec01f2a838)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-15 13:15:59 +05:30
Marcus Sorensen
8597758ed8 Support Jetty's live cert reload on HTTPS frontend (#7355)
* Support Jetty's live cert reload

Signed-off-by: Marcus Sorensen <mls@apple.com>

* Update ServerDaemon.java

---------

Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
(cherry picked from commit 9ca5f287eb4ad6c136fa49a8f752ff0a535bf088)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-15 13:15:59 +05:30
Rohit Yadav
c8f02a8b34 engine/orchestartion: fix/refactor to Account.Type.ADMIN
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-15 13:15:59 +05:30
Marcus Sorensen
0c192f3609 When VM start fails at host for admin, report error (#208)
* When VM start fails at host for admin, report error

Signed-off-by: Marcus Sorensen <mls@apple.com>

* Report ResourceUnavailableExceptions that result in InsufficientCapacityException to admin

* Update error message to be more straightforward

Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
(cherry picked from commit 8885d252f07a2dc0d46b3d1d4e09d919e2599787)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-14 20:00:46 +05:30
Wei Zhou
4bdff06acd Updating pom.xml version numbers for release 4.18.1.0
Signed-off-by: Wei Zhou <weizhou@apache.org>
4.18.1.0
2023-09-07 08:50:50 +02:00
Abhishek Kumar
f049f5409e
server: fix dualstack ipv6 networks for vxlan (#7933)
Fixes #7926

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-09-07 08:46:45 +02:00
Wei Zhou
126dd5fa4c
kvm: fix live vm migration between local storage pools (#7945) 2023-09-07 08:22:37 +05:30
Rohit Yadav
0bb05f7871
ui: don't skip if dataview has multiple items in response (#7947)
This would fix the case of multiple items return in API response for a
resource such as a template or ISO in case of multi-zone env.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-06 16:41:41 +02:00
Nicolas Vazquez
57c61fb33c
Fix direct download https compressed qcow2 template checker (#7932)
This PR fixes an issue on direct download while registering HTTPS compressed files
Fixes: #7929
2023-09-01 08:16:03 +02:00
Marcus Sorensen
89e0a4ca25
PowerFlex: Handle missing volumes gracefully during delete volume (#7924) 2023-08-31 21:43:45 +02:00
Marcus Sorensen
2cccd8f754
db: Add index on cluster_details.name for FirstFitPlanner speedup (#7922) 2023-08-31 21:43:23 +02:00
Wei Zhou
439d70fd2b
xen/xcpng: set vmr.VCPUsMax to minimum of global setting and host cpu cores (#7887)
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-08-28 14:26:20 +02:00
Wei Zhou
26581b7741
server: throw new exception when rootdisksize is required but not set (#7913)
* server: throw new exception when rootdisksize is required but not set

* PR7913: fix an issue with PR6441
2023-08-28 08:57:40 +02:00
Wei Zhou
0dd6bb7c67
systemvm: update size and state of systemvm template during registration (#7874) 2023-08-25 16:46:32 +02:00
Fabricio Duarte
57d4d0d94a
UI: Infer template settings in the deploy VM wizard (#7867) 2023-08-25 16:36:35 +02:00
Rohit Yadav
93bd5b7a69
ui: speed up compute instance listing (#7911)
* ui: speed up compute instance listing

The default listVirtualMachinesMetrics APIs assumings details=all which
isn't really used in the list view. By changing this to a subset of
details, we can see gains upto 10x in listing speed in the UI. When
moving to the resource/detail view of the UI, don't pass state or
details so `all` the details of the instance are returned.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* Update AutogenView.vue

---------

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-25 13:57:44 +02:00
Rohit Yadav
6f7725ab35
ui: Fix refresh and re-route behaviour (#7846)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-08-25 13:45:35 +02:00
Wei Zhou
b19db10ea6
server: try with all supported hypervisors (ordered, no duplications) when deploy a virtual router (#7877) 2023-08-25 11:49:03 +02:00
Wei Zhou
8dc5fdd067
server: fix cannot get systemvm ips in dedicated ranges (#7144)
This fixes #6698
2023-08-25 11:36:39 +02:00
Wei Zhou
f5a1f4130d
server: fix global setting system.vm.public.ip.reservation.mode.strictness is not really dynamic (#7909)
If the original value is `false`, and search build is configured without the condition. Now change the value to `true`, it will not get effective due to missing condition.
2023-08-25 11:35:31 +02:00
Rohit Yadav
e964395bd4
vmware: improve solidfire storage plugin integration and fix cases (#3) (#7761)
This fixes the following cases in which Solidfire storage integration
caused issues when using Solidfire datadisks with VMware:

1. Take Volume Snapshot of Solidfire data disk
2. Delete an active Instance with Solidfire data disk attached
3. Attach used existing Solidfire data disk to a running/stopped VM
4. Stop and Start an instance with Solidfire data disks attached
5. Expand disk by resizing Solidfire data disk by providing size
6. Expand disk by changing disk offering for the Solidfire data disk

Additional changes:
- Use VMFS6 as managed datastore type if the host supports
- Refactor detection and splitting of managed storage ds name in storage
  processor
- Restrict storage rescanning for managed datastore when resizing

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-25 11:36:03 +05:30