1062 Commits

Author SHA1 Message Date
John Bampton
11d45654a6
misc: fix spelling (#7206)
This PR fixes spellings
2023-05-23 11:06:16 +05:30
John Bampton
c2e17310d6
Add three more pre-commit checks (#7083)
Co-authored-by: dahn <daan@onecht.net>
2023-03-27 13:28:55 +02:00
Daan Hoogland
1d0762e43a Merge release branch 4.18 to main
* 4.18:
  Fix PR 7131 bugs and vulnerabilities (#7140)
2023-03-23 15:54:19 +01:00
João Jandre
523ab58d02
Fix PR 7131 bugs and vulnerabilities (#7140) 2023-03-21 15:06:18 +01:00
Daan Hoogland
fb4f6a334d Updating pom.xml version numbers for release 4.19.0.0-SNAPSHOT
Signed-off-by: Daan Hoogland <daan@onecht.net>
2023-03-15 19:46:01 +01:00
Daan Hoogland
05cda2729f Updating pom.xml version numbers for release 4.18.1.0-SNAPSHOT
Signed-off-by: Daan Hoogland <daan@onecht.net>
2023-03-15 19:38:14 +01:00
Daan Hoogland
0574087284 Updating pom.xml version numbers for release 4.18.0.0
Signed-off-by: Daan Hoogland <daan@onecht.net>
2023-03-11 09:35:41 +01:00
SadiJr
f5b3cb59ee
[Veeam] enable volume attach/detach in VMs with Backup Offerings (#6581) 2023-01-23 09:34:46 +01:00
SadiJr
d04d60b079
[VMWare] Limit IOPS in Compute/Disk Offerings (#6386) 2023-01-17 14:41:56 +01:00
Eduardo Zanetta
a9b49f3ae9
Cleanup APIs getCommandName (#7022)
Co-authored-by: Eduardo Zanetta <eduardo.zanetta@scclouds.com.br>
2023-01-03 12:11:52 +01:00
John Bampton
def7ce655d
Fix spelling (#6898)
Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>
2022-12-13 14:58:14 +01:00
Wei Zhou
74623aa3d3 Merge remote-tracking branch 'apache/4.17' 2022-12-07 13:37:30 +01:00
Abhishek Kumar
9fd2d82af9
vmware: fix vmware vm snapshot operations for datastorecluster (#6912)
Fixes #6894

Fixes NPE observed when VM volume is present on the same child datastore.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-12-07 14:43:44 +05:30
Wei Zhou
a63b2aba7a
VM Autoscaling with virtual router (#6571) 2022-12-05 15:23:03 +01:00
John Bampton
e65c22d883
Fix spelling (#6860) 2022-11-13 10:56:15 +01:00
John Bampton
fa39e61a4c
Fix spelling (#6609) 2022-11-01 08:30:01 +01:00
Harikrishna
713a236843
UserData as first class resource (#6202)
This PR introduces a new feature to make userdata as a first class resource much like existing SSH keys.

Detailed feature specification document:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Userdata+as+a+first+class+resource
2022-10-05 17:34:59 +05:30
João Jandre
efbf74ee06
Added new logs to volume creation (#6689)
Co-authored-by: João Paraquetti <joao@scclouds.com.br>
2022-09-26 19:11:14 -03:00
Abhishek Kumar
d724a9d15c Updating pom.xml version numbers for release 4.17.2.0-SNAPSHOT
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-09-19 16:21:35 +05:30
Abhishek Kumar
350ef38e1c Updating pom.xml version numbers for release 4.17.1.0
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-09-14 12:58:03 +05:30
Abhishek Kumar
e720b72e15 Merge remote-tracking branch 'apache/4.17' into main 2022-08-31 17:38:30 +05:30
Abhishek Kumar
a21efe75df
vmware: fix vm snapshot with datastore cluster, drs (#6643)
Fixes #6595
Sync volume datastore, path and chaininfo info while calculating snapshot chain size after snapshot operation is complete from vCenter.
2022-08-31 16:00:14 +05:30
Abhishek Kumar
7ae6fdf053 Merge remote-tracking branch 'apache/4.17' into main 2022-08-16 17:51:09 +05:30
Abhishek Kumar
770df60bbe
vmware,cks: fix attachiso failure with vmware drs (#6625)
Fixes #4314

Failure in attaching k8s ISO is seen when VMware DRS is enabled. Log reported VM is not found. This fix tries to find VM on peer hosts when the VM is not found on the given host.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-08-16 16:17:13 +05:30
dahn
731a83babf
add global setting to allow parallel execution on vmware (#6413)
* add global setting to allow parallel execution on vmware

* cleanup setting distribution for vmware.create.full.clone

* query setting in vmware guru

* don´t touch other hypervisor's commands

* guru hierarchy cleanup
2022-07-15 10:01:35 +02:00
Daan Hoogland
a470f3353a Merge branch '4.17' 2022-07-05 09:11:45 +02:00
João Jandre
e467ad02ce
Fix VMware memory retrieval (#6414)
* Fix VMware memory retrieval

* Address review

Co-authored-by: Joao <JoaoJandre@gitlab.com>
2022-07-01 23:36:40 -03:00
Rohit Yadav
6607a98597 Merge remote-tracking branch 'origin/4.17' 2022-07-01 10:44:29 +05:30
Harikrishna
637a102922
Fix for VMware VM migration with volume in local storage (#6483)
* Fix VMware VM migration with volume in case of local storage

* Break the loop once target host is found

* Code optimisations in getting the target host guid for local storage

* Fixed code smells and added unit test
2022-07-01 10:39:02 +05:30
nvazquez
0bcc609f05
Updating pom.xml version numbers for release 4.18.0.0-SNAPSHOT
Signed-off-by: nvazquez <nicovazquez90@gmail.com>
2022-06-06 12:25:35 -03:00
nvazquez
038a669d6b
Updating pom.xml version numbers for release 4.17.1.0-SNAPSHOT
Signed-off-by: nvazquez <nicovazquez90@gmail.com>
2022-06-06 12:19:44 -03:00
nvazquez
c56220fcf2
Updating pom.xml version numbers for release 4.17.0.0
Signed-off-by: nvazquez <nicovazquez90@gmail.com>
2022-05-31 14:33:47 -03:00
Pearl Dsilva
3bd723ca06
Change patch path during live patching of systemVMs (#6328)
* Improve log when live patching fails

* change patching path from /tmp to /var/cache/clou

* add iptable rule for console proxy (novnc)

* temporary template paths

* revert pom xml to original paths
2022-04-29 00:01:11 -03:00
SadiJr
4cc830d04a
[VMWare] error when detaching volume (#6283)
* Fix exception, in VMWare 6.5, when trying to detach volume

* Address reviews

* Address reviews

Co-authored-by: SadiJr <sadi@scclouds.com.br>
2022-04-26 23:29:17 -03:00
Pearl Dsilva
830f3061bc
SystemVM optimizations (#5831)
* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* refactor patching logic - certs

* move logic of fixing rootdiskcontroller from upgrade to kubernetes service

* add livepatch option to restart network & vpc

* smooth upgrade of cks clusters

* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* add cgroup config for containerd

* add systemd config for kubelet

* add additional info during image registry config

* address comments

* add temp links of download.cloudstack.org

* address part of the comments

* address comments

* update containerd config - as version has upgraded to 1.5 from 1.4.12 in 4.17.0

* address comments - simplify

* fix vue3 related icon changes

* allow network commands when router template version is lower but is patched

* add internal LB to the list of routers to be patched on network restart with live patch

* add unit tests for API param validations and new helper utilities - file scp & checksum validations

* perform patching only for non-user i.e., system VMs

* add test to validate params

* remove unused import

* add column to domain_router to display software version and support networkrestart with livePatch from router view

* Requires upgrade column to consider package (cloud-scripts) checksum to identify if true/false

* use router software version instead of checksum

* show N/A if no software version reported i.e., in upgraded envs

* fix deb failure

* update pom to official links of systemVM template
2022-04-21 13:40:19 -03:00
DK101010
ccac1a383f
Feat/add vdisk UUID to list volume (#5848)
* get vdisk uuid from vcenter and store it into database

* add vdisk uuid as external_uuid to listVolume response

* add sql upgrade file

* Update vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* update sql add column external_uuid

* Update server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java

Co-authored-by: Wei Zhou <weizhou@apache.org>

* adapt param description for externalUuid

* add 'idempotent column add' to create external_uuid col

* rename method to getExternalDiskUUID

* remove line disk_offering.system_use

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
2022-04-19 23:34:09 -03:00
John Bampton
c265df965e
Fix spelling (#6185)
* Fix spelling

* Update services/secondary-storage/server/src/main/java/org/apache/cloudstack/storage/resource/NfsSecondaryStorageResource.java

Co-authored-by: SadiJr <sadijacinto@gmail.com>

Co-authored-by: Rohit Yadav <rohityadav89@gmail.com>
Co-authored-by: SadiJr <sadijacinto@gmail.com>
2022-04-16 00:55:11 +05:30
José Flauzino
16f2896940
Persistence of VM stats (#5984)
* Add persistence of VM stats

* Fix API 'since' attribute

* Add license

* Address GutoVeronezi's reviews

* Fix the order of VM stats in the API response

* Fix msid in VM stats data

* Fix disk stats and add minor improvements

* Add log message

* Build string using ReflectionToStringBuilderUtils

* Rerun checks

Co-authored-by: joseflauzino <jose@scclouds.com.br>
2022-04-11 10:42:21 -03:00
David Jumani
5ab4309298
agent: Detect existing hosts with UEFI support (#6139)
* agent: Pass uefi enabled status as part of ready command

* Cleanup

* Fix checkstyle

* Save uefi status if different
2022-04-04 23:31:54 -03:00
JoaoJandre
5f07ddaca9
Refactor account type (#6048)
* Refactor account type

* Added license.

* Address reviews

* Address review.

Co-authored-by: João Paraquetti <joao@scclouds.com.br>
Co-authored-by: Joao <JoaoJandre@gitlab.com>
2022-03-09 11:14:19 -03:00
John Bampton
6401c850b7
Fix spelling (#6064)
* Fix spelling

- `interupted` to `interrupted`
- `paramter` to `parameter`

* Fix more typos
2022-03-08 13:02:35 -03:00
dahn
b014617416
no axis (#5993)
* no axis

* remove f5 from ui

Co-authored-by: Daan Hoogland <dahn@onecht.net>
2022-03-07 22:46:29 -03:00
John Bampton
980c0bff03
Fix spelling (#6041)
Changed `occured` to `occurred`
2022-03-03 13:10:14 -03:00
Pearl Dsilva
550c810538
[VMware] Support for removal of NIC on IP disassociation on the VR (#5985)
* [VMware] Support for removal of NIC on IP disassociation on the VR

* address comments - extract code + add appropriate operation name in logs

* Do not remove/unplug nic on removal of static nat on last IP (Public NIC)

* address comments

* address comment
2022-02-17 22:44:03 -03:00
Suresh Kumar Anaparti
212a4dedfc
Merge branch '4.16' into main 2022-02-10 20:56:52 +05:30
Nicolas Vazquez
a3bb84b495
[VMware] Update SCSI controllers for VMs (#5910) 2022-02-10 14:36:34 +01:00
Suresh Kumar Anaparti
e3f0e3b544
Merge branch '4.16' into main 2022-02-04 19:04:08 +05:30
Harikrishna
56a8852c4c
Remove code that reserves extra interfaces based on the global setting router.extra.public.nics in case of VMware. (#5912)
Added hot plugging of vifs whenever required in case of isolated networks. VPC networks already uses hotplugging of vifs.
2022-02-04 12:41:56 +01:00
Harikrishna
f15cab16da
server: Decouple service (compute) offering and disk offering (#5008)
Currently, our compute offerings and disk offerings are tightly coupled with respect to many aspects. For example, if a compute offering is created, a corresponding disk offering entry is also created with the same ID as the reference. Also creating compute offering takes few disk-related parameters which anyway goes to the corresponding disk offering only. I think this design was initially made to address compute offering for the root volume created from a template. Also changing the offering of a volume is tightly coupled with storage tags and has to be done in different APIs either migrateVolume or resizeVolume. Changing of disk offering should be seamless and should consider new storage tags, new size and place the volume in appropriate state as defined in disk offering.

more details are mentioned here https://cwiki.apache.org/confluence/display/CLOUDSTACK/Compute+offering+and+disk+offering+refactoring

* Schema changes and disk offering column change from "type" to "compute_only"

* Few more changes

* Decoupled service offering and disk offering

* Remove diskofferingid from vminstance VO

* Decouple service offering and disk offering states

* diskoffering getsize() is only for strict disk offerings

* Fix deployVM flow

* Added new API params to compute offering creation

* Add diskofferingstrictness to serviceoffering vo under quota

* Added overrideDiskOfferingId parameter in deploy VM API which will override disk offering for the root disk both in template and ISO case

Added diskSizeStrictness parameter in create Disk offering API which will decide whether to restrict resize or disk offering change of a volume

* Fix User vm response to show proper service offering and disk offerings

* Added disk size strictness in disk offering response

* Added disk offering strictness to the service offering response

* Remove comments

* Added UI changes for Disk offering strictness in add compute offering form and Disk size strictness in add disk offering form

* Added diskoffering details to the service offering response

* Added UI changes in deployvm wizard to accept override disk offering id

* Fix delete compute offering

* Fix VM deployment from custom service offering

* Move uselocalstorage column access from service offering to disk offering

* UI: Separated compute and disk releated parameters in add compute offering wizard, also added association to disk offering

* Fixed diskoffering automatic selection on add compute offering wizard

* UI: move compute only toggle button outside the box in add compute offering wizard

* Added volumeId parameter to listDiskOfferings API and the disksizestrictness flag of the current disk offering is honored while list disk offerings

* Added configuration parameter to decide whether to check volume tags on the destination storagepool during migration

* Added disk offering change checks during resize volume operation

* Added new API changeofferingforVolume API and corresponding changes

* Add UI form for changeOfferingForVolume API

* Fix UI conflicts

* Fix service offering usage as disk offering

* Fix unit test failures

* fix user_vm_view

* Addressed review comments

* Fixed service_offering_view

* Fix service offering edit flow

* Fix service offering constructor to address custom offering

* Fix domain_router_view to get proper service offering id

* Removed unused import

* Addressed review comments and fixed update service offering flow with storage tags

* Added marvin test cases for checking disk offering strictness

* review comments addressed

* Remove system_use column from disk offering join

* update volume_view to update system_use column from service offering and not disk offering

* Fix changeOfferingForVolume API for custom disk offering

* Fix global setting implementation

* Fix list volumes, after changing system_use column from disk offering to service offering in volume_view

* Changes for override root disk offering in deployvm wizard in case of custom offering

* Fix a unit test case

* Fixed recent unit test cases with new serviceofferingvo constructor

* Fix unit test in VolumeApiServiceImpl

* Added storage id for the list disk offering API and corresponding UI changes in migrateVolume and changeOfferingForVolume flow

* Rename global configuration parameter from storage.pool.tags.disk.offering.strictness to match.storage.pool.tags.with.disk.offering

* Fix smoke test failures

* Added tool tip for migrate volume UI form

* Address review comments and fix UI form of deploy VM in case of ISO.

* Fixed resize volume UI form for data disk

* UI changes to disable override root disk size when override root disk offering is enabled

* UI fix in deploy vm wizard

* Fix listdiskoffering after rebasing with main

* Fixed UI in migrate and changeofferingfor volume to handle empty disk offering list
Removed the volume's current disk offering from listDiskOffering response list

* Added custom Iops to resize volume form and removed the current disk offering during change offering for volume UI form

* Fix false response on updateDiskOffering API

* Added search field for changeofferingforvolume UI form

* Fix resize volume and migrate volume to update volume path if DRS is applied on volume in datastore cluster

* Removed DB changes from 4.16 upgrade file

* Resolving merge conflicts with main 4.17

* Added support for auto migration and auto resize of the root volume upon changing the service offering for VM.

* UI: Added automigrate checkbox in scale VM form

* Addes since attributes to new API params

* Added shrinkOK parameter to changeofferingforvolume API

* Added shrinkOk param to UI in changeOfferingforVolume form

* Added shrinkOk flag to scaleVM and changeServiceForVirtualMachines and UI form

* Removed old foreign key constraint on IDs of service offering and disk offering

* Allow resize and automigrate of root volume if required in all cases of service offering change

* Allow only resize to higher disk size from UI

* Fixing vue syntax error

* Make UI changes to provide root disk size box when the linked disk offering is of custom

* Converted from check box to toggle in scale VM, changeoffering, resize and migrate volume forms

* Fix resize volume operation to update the VM settings

* Fix migratevolume form to pick selected storage pool id in list diskofferings API
2022-01-27 15:08:42 +05:30
SadiJr
5d7ea30f87
vmware: Remove restriction, in VMWare, to resize only SCSI disks (#5650)
Remove restriction, in VMWare, to resize only SCSI disks, because VMWare supports resize of subtypes of SCSI, like LSILOGIC, LSILOGIC SAS, etc

In VMWare, there are different types of SCSI controllers, such as BusLogic Parallel, LSI Logic Parallel, LSI Logic SAS, etc. All of these sub-types support the disk resize operation, the only exception is the IDE controller. However, in ACS, there is a logical restriction to allow only resizing of volumes of the specific SCSI type; thus, not allowing resizing of the SCSI sub-types. This PR corrects this behavior blocking the resize of volumes which uses IDE controller only.
More information can be found at:

https://www.starwindsoftware.com/blog/storage-controllers-in-vmware-vsphere
https://www.nakivo.com/blog/scsi-controller-and-other-vmware-controller-types/
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-5872D173-A076-42FE-8D0B-9DB0EB0E7362.html
https://communities.vmware.com/t5/ESXi-Discussions/Unable-to-increase-disk-Space/td-p/397823

Co-authored-by: SadiJr <sadi@scclouds.com.br>
2022-01-08 09:02:00 +05:30