3394 Commits

Author SHA1 Message Date
Suresh Kumar Anaparti
620ed164d8
VMware: Improve error messaging / logs when starting non-user VMs, and secondary storage not available or doesn't have enough capacity (#9207) 2024-06-25 12:25:42 +05:30
Rene Glover
6ee6603359
Updates to HPE-Primera and Pure FlashArray Drivers to use Host-based VLUN Assignments (#8889)
* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* improve error handling when copying volumes to add precision to which step failed

* rename pure volume before delete to avoid conflicts if the same name is used before its expunged on the array

* remove dead code in AdaptiveDataStoreLifeCycleImpl.java

* Fix issues found in PR checks

* fix session refresh TTL logic

* updates from PR comments

* logic to delete by path ONLY on supported OUI

* fix to StorageSystemDataMotionStrategy compile error

* change noisy debug message to trace message

* fix double callback call in handleVolumeMigrationFromNonManagedStorageToManagedStorage

* fix for flash array delete error

* fix typo in StorageSystemDataMotionStrategy

* change copyVolume to use writeback to speed up copy ops

* remove returning PrimaryStorageDownloadAnswer when connectPhysicalDisk returns false during KVMStorageProcessor template copy

* remove change to only set UUID on snapshot if it is a vmSnapshot

* reverting change to UserVmManagerImpl.configureCustomRootDiskSize

* add error checking/simplification per comments from @slavkap

* Update engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* address PR comments from @sureshanaparti

---------

Co-authored-by: GLOVER RENE <rg9975@cs419-mgmtserver.rg9975nprd.app.ecp.att.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-25 10:35:39 +05:30
Abhishek Kumar
4d6df586da
engine-schema: fix options for host.allocators.order config (#8733)
With d8c7e34b38 options were added to the host.allocators.order config. Currently, it allows adding only FirstFitRouting as the value. This PR fixes the behaviour and allows other host allocators to be added.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-24 15:55:48 +05:30
Abhishek Kumar
de266a4cb8
api,server: allow updating hypervisor capabilities with hypervisor and version (#8475)
* api,server: allow updating hypervisor capabilities with hypervisor and
version

hypervisor and hypervisorversion parameter added to the
updateHypervisorCapabilities API.

* param description

* Update server/src/main/java/com/cloud/server/ManagementServerImpl.java

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Henrique Sato <henriquesato2003@gmail.com>
Co-authored-by: Vishesh <vishesh92@gmail.com>
2024-06-24 14:41:01 +05:30
João Jandre
3e30283500
Fix migration from local storage to NFS in KVM (#8909)
* fix migration from local to nfs

* remove unused imports

* remove dead code
2024-06-24 13:02:48 +05:30
Pearl Dsilva
f792684b9c
Support migration of VM imported from a remote host (#9259) 2024-06-24 12:46:21 +05:30
Suresh Kumar Anaparti
9055610034
Remove duplicate network state checks before shutdown network (#8462) 2024-06-21 14:12:07 +02:00
Wei Zhou
0c422aca76
server: fix additional zones cannot be removed (#9261)
Got an exception when delete a zone

```
com.cloud.utils.exception.CloudRuntimeException: The zone cannot be deleted because there are Secondary storages in this zone
```
2024-06-21 15:44:30 +05:30
João Jandre
ae3fa5d0de
Add configuration to limit the number of rows deleted from vm_stats (#8740)
Co-authored-by: João Jandre <joao@scclouds.com.br>
2024-06-20 14:26:36 +02:00
Harikrishna
2315a73a20
User friendly name of Downloaded Templates Volumes and ISOs (#9252) 2024-06-19 12:47:43 +02:00
Suresh Kumar Anaparti
8b02624e61
User data content size validation, and related code improvements (#8418)
Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-06-18 20:07:13 +02:00
Daan Hoogland
050ee44137 Merge branch '4.18' into 4.19 2024-06-18 16:05:45 +02:00
Bryan Lima
a10eee25b3
Add method for decrypting values accordingly (#9088) 2024-06-18 10:31:35 +02:00
dahn
7c5b7ca077
Extra parameter for UpdateImageStore (#8941)
* Extra parameter for UpdateImageStore

* add name parameter

* ui

* cleanup

* update DB from storage stats results
2024-06-18 12:31:17 +05:30
Harikrishna
bb0c1f93af
Add volume encryption checks during the disk offering change (#9209) 2024-06-17 10:36:47 +02:00
dahn
ed86dc973b
protect against missing service offering (#9235)
* protect agains missing service offering

* search removed before assuming none

* import

* javadoc
2024-06-14 15:25:46 +05:30
Rohit Yadav
2ca0857bd5
api: listVM API improvement followup, change returning of stats detail (#9177)
- Changes behaviour of details param handling via global setting:
  - listVirtualMachines API: when the details param is not provided, it returns whether stats are returned controlled by a new global setting `list.vm.default.details.stats`
  - listVirtualMachinesMetrics API: when the details param is not provided, it uses `all` details including `stats`
- Users who are affected slow performance of the listVirtualMachines API response time can set `list.vm.default.details.stats` to `false`
- Remove ConfigKey vm.stats.increment.metrics.in.memory which was renamed to `vm.stats.increment.metrics` in #5984 and also remove unused/unnecessary global settings via upgrade path
- Changes default value of VM stats accumulation setting `vm.stats.increment.metrics` to false until a better solution emerges. Since #5984, this is true and during the execution of listVM APIs the stats are clubbed/calculated which can immensely slow down list VM API calls. Any costly operations such as summing of stats shouldn't be done during the course of a synchronous API, such as the list VM API.
- Fix UI that uses listVirtualMachinesMetrics to not call `stats` detail when in list view without metrics selected.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-06-14 11:03:14 +05:30
Abhishek Kumar
abbc61c01e
engine-orchestration: expunge destroyed system vm volume (#9197)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-13 10:00:22 +02:00
Rohit Yadav
0f8a839a1d
engine/schema: force index in user_vm_view to speed up VM instance listing (#9198)
The user_vm_view can end up not picking the right index to join against
the user_ip_address table causing full table scan on the user_ip_address
table. This could be related to a MySQL bug
https://bugs.mysql.com/bug.php?id=41220

In a test environment with 20k shared networks and over 20M IPs, the
listVirtualMachines API was found to take over 17s to return list of
just 10 VMs. However, with this fix it would now take under 200ms to
return the list.

MySQL slow query logging showed ~nearly 20M table scans of the IP
address table:
```
# User@Host: cloud[cloud] @ localhost [127.0.0.1]  Id:    39
# Query_time: 8.227541  Lock_time: 0.000014 Rows_sent: 12  Rows_examined: 19,667,235
SET timestamp=1715410270;
SELECT user_vm_view.id, user_vm_view.name /*snipped*/ FROM user_vm_view
WHERE user_vm_view.id IN (4,6,7,8,9,10,11,12,13,14,15,16);
```

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-06-13 12:29:11 +05:30
Wei Zhou
b2ef53b8a2
kvm: replace ISO path in vm XML configuration during vm migration (#9212)
* kvm: replace ISO path in vm XML configuration during vm migration

* Update 9212: address comments

* kvm: fix vm migration if there are multiple image stores
2024-06-12 16:01:23 +02:00
dahn
4de975ff25
list by isEncrypted (#8643)
* list by isEncrypted

* use filter on VO and cleanup

* add encryption type to volume response

* Update api/src/main/java/org/apache/cloudstack/api/command/user/volume/ListVolumesCmd.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-12 15:11:00 +05:30
Abhishek Kumar
fcca3e8f39
api,server: encryptformat in volume response (#9106)
Fixes #9014

* api,server: encryptformat in volume response

* Apply suggestions from code review

* remove passphrase db change
2024-06-11 11:20:50 +05:30
Abhishek Kumar
91c7bc722f
server,cks: check if vm is cks node during vm destroy (#9057)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-06 10:54:02 +02:00
Abhishek Kumar
c970141e25
engine-schema: add removed columne for cloud.user_data, fix delete op (#9120)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-05-28 09:43:14 +02:00
Abhishek Kumar
daf6b9d103
api,ui: vm template format, fix vm info link (#9094) 2024-05-24 17:39:52 +02:00
Vishesh
6b4955affe
Fix message publish in transaction (#8980)
* Fix message publish in transaction

* Resolve comments
2024-05-07 13:27:31 +05:30
dahn
e520525fe7
Use parameter dcId as wrapper to prevent NPE (#8986) 2024-05-01 09:12:36 +02:00
Rohit Yadav
3de1f8b4ba Merge remote-tracking branch 'origin/4.18' into 4.19
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-04-29 13:44:34 +05:30
Vishesh
80a8b80a9d
Update volume's passphrase to null if diskOffering doesn't support encryption (#8904) 2024-04-29 12:18:09 +05:30
Wei Zhou
0e08a126df
systemvm: add template_zone_ref record when add a new zone with same hypervisor type (#8395) 2024-04-25 08:41:59 +02:00
SadiJr
96ae479000
[Usage] Create network billing (#7236)
Co-authored-by: Bryan Lima <bryan.lima@hotmail.com>
Co-authored-by: SadiJr <sadi@scclouds.com.br>
Co-authored-by: Bryan Lima <42067040+BryanMLima@users.noreply.github.com>
Co-authored-by: Henrique Sato <henriquesato2003@gmail.com>
2024-04-24 08:52:49 +02:00
Wei Zhou
0b857def68
New feature: Import/Unamange DATA volume from storage pool (#8808) 2024-04-23 16:05:59 +02:00
dahn
6502dde8c3
field enlarged and db upgrade (#8675)
* 4.19 -> 4.19.1 and enlarge url field
2024-04-23 13:01:08 +05:30
SadiJr
49d244f251
[Usage] Create VPC billing (#7235)
Co-authored-by: SadiJr <sadi@scclouds.com.br>
Co-authored-by: Bryan Lima <bryan.lima@hotmail.com>
2024-04-22 09:42:24 +02:00
João Jandre
8a101fbbc1 Updating pom.xml version numbers for release 4.18.3.0-SNAPSHOT
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-04-17 11:11:57 -03:00
Vishesh
44aa08c02a
Fixup 4.19 build issue (#8905) 2024-04-12 16:37:25 +02:00
Vishesh
b998e7dbb6
Allow overriding root disk offering & size, and expunge old root disk while restoring a VM (#8800)
* Allow overriding root diskoffering id & size while restoring VM

* UI changes

* Allow expunging of old disk while restoring a VM

* Resolve comments

* Address comments

* Duplicate volume's details while duplicating volume

* Allow setting IOPS for the new volume

* minor cleanup

* fixup

* Add checks for template size

* Replace strings for IOPS with constants

* Fix saveVolumeDetails method

* Fixup

* Fixup UI styling
2024-04-12 17:47:52 +05:30
João Jandre
154566f914 Updating pom.xml version numbers for release 4.18.2.0
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-04-12 08:25:04 -03:00
Vishesh
730cc5d5b8
Change iops on offering change (#8872)
* Change IOPS on disk offering change

* Remove iops & bandwidth limits before copying template

* minor refactor

* Handle diskOfferingDetails

* Fixup
2024-04-11 17:01:55 +05:30
Vishesh
c24c1a5c00
Fix VM import & VM delete with custom offering (#8813) 2024-04-10 11:29:29 +02:00
Vishesh
b8f8321f0f
Allow listing of inactive offerings (#8821) 2024-04-04 13:46:44 +02:00
Rohit Yadav
f2da882d68 Merge remote-tracking branch 'origin/4.18' into 4.19 2024-04-04 14:58:50 +05:30
Wei Zhou
7a9985b42b
upgrade: add unit tests from/to a security release (#8870)
* upgrade: add unit tests from/to a security release

* upgrade: add unit test from a old security release to a new security release
2024-04-04 14:55:29 +05:30
Abhishek Kumar
ff3e9bd821 engine-storage: control download redirection
Add a global setting to control whether redirection is allowed while
downloading templates and volumes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-04-04 14:11:05 +05:30
Wei Zhou
939d0b9011 engine-storage: control download redirection
Add a global setting to control whether redirection is allowed while
downloading templates and volumes

core: some changes on SimpleHttpMultiFileDownloader
similar as HttpTemplateDownloader

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
(cherry picked from commit b1642bc3bf58ccde9f56f632b5a9fe46a3eb5356)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-04-04 11:19:20 +05:30
Wei Zhou
fc3c625beb server: fix security issues caused by extraconfig on KVM
- Move allow.additional.vm.configuration.list.kvm from Global to Account setting
- Disallow VM details start with "extraconfig" when deploy VMs
- Skip changes on VM details start with "extraconfig" when update VM settings
- Allow only extraconfig for DPDK in service offering details
- Check if extraconfig values in vm details are supported when start VMs
- Check if extraconfig values in service offering details are supported when start VMs
- Disallow add/edit/update VM setting for extraconfig on UI

(cherry picked from commit e6e4fe16fb1ee428c3664b6b57384514e5a9252e)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 7aea9db1c8d8ef1febb723913dc01c3040641425)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-04-04 11:19:08 +05:30
Wei Zhou
72b2eb0087 server: fix security issues caused by extraconfig on KVM
- Move allow.additional.vm.configuration.list.kvm from Global to Account setting
- Disallow VM details start with "extraconfig" when deploy VMs
- Skip changes on VM details start with "extraconfig" when update VM settings
- Allow only extraconfig for DPDK in service offering details
- Check if extraconfig values in vm details are supported when start VMs
- Check if extraconfig values in service offering details are supported when start VMs
- Disallow add/edit/update VM setting for extraconfig on UI

(cherry picked from commit e6e4fe16fb1ee428c3664b6b57384514e5a9252e)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-04-04 11:00:10 +05:30
Marcus Sorensen
2e88eb45a3
Update mysql-connector version (#8753)
Co-authored-by: Marcus Sorensen <mls@apple.com>
2024-03-21 18:09:06 +05:30
Vishesh
24d5c45f7f
Add indexes for vm_stats (#8737)
* Add indexes for vm_stats

* Remove index on timestamp

* Chnage index from vm_id,timestamp to vm_id
2024-03-21 14:05:47 +05:30
Vishesh
0043540fa3
Use join instead of views (#8321) 2024-03-18 18:08:19 +01:00