990 Commits

Author SHA1 Message Date
Nicolas Vazquez
2f2d6cbe38
KVM: Enhance CPU speed detection on hosts (#6175)
* KVM: Enhance CPU speed detection on hosts

* Update agent/conf/agent.properties

Co-authored-by: dahn <daan.hoogland@gmail.com>

Co-authored-by: dahn <daan.hoogland@gmail.com>
2022-04-01 12:12:34 -03:00
Wei Zhou
944b6037cc Merge remote-tracking branch 'apache/4.16' 2022-03-29 10:43:59 +02:00
Nicolas Vazquez
a69ab3b28f
Ensure configdrive path is edited properly during live migration (#6173) 2022-03-29 10:09:08 +02:00
Wei Zhou
5845de4004
kvm: support multiple local storage pools (#6147)
* kvm: support multiple local storage pools

* Update #6147: add method configureLocalStorage

* Update #6147: add method validateLocalStorageUUID

* Update #6147: remove underscore

* Update #6147: add unit tests
2022-03-25 11:40:30 -03:00
Bryan Lima
afdc73f911
Update VM priority (cpu_shares) when live scaling it (#6031)
* Update VM priority (cpu_chares) when live scaling it

* Apply suggestions from code review

Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Addressing javadoc review

* Addressing review typo

Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2022-03-14 11:34:10 -03:00
John Bampton
6401c850b7
Fix spelling (#6064)
* Fix spelling

- `interupted` to `interrupted`
- `paramter` to `parameter`

* Fix more typos
2022-03-08 13:02:35 -03:00
Rohit Yadav
2f6fc56e74
maven: Fix build on osx (#6056)
kvm: don't read /proc/meminfo on non-Linux environments as part of constructor

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
2022-03-07 14:55:42 +05:30
John Bampton
980c0bff03
Fix spelling (#6041)
Changed `occured` to `occurred`
2022-03-03 13:10:14 -03:00
Suresh Kumar Anaparti
bc70535ee5
Updating pom.xml version numbers for release 4.16.2.0-SNAPSHOT
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2022-03-03 18:15:33 +05:30
Suresh Kumar Anaparti
cad9332082
Updating pom.xml version numbers for release 4.16.1.0
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2022-02-25 19:01:16 +05:30
Suresh Kumar Anaparti
1484f5a418
Merge branch '4.16' into main 2022-02-23 22:42:10 +05:30
Suresh Kumar Anaparti
027e6030af
[KVM] Disconnect the volumes with the proper storage adaptor. (#6029)
* [KVM] Disconnect the volumes with the proper storage adaptor.

* Improved / Added logs
2022-02-23 22:40:14 +05:30
Suresh Kumar Anaparti
850714c1fc
Merge branch '4.16' into main 2022-02-10 10:57:29 +05:30
dahn
143b72d67e
cleanup: Network Throttling for Additional Networks code in DirectVifDriver.java (#5875) 2022-02-09 10:19:47 +01:00
Suresh Kumar Anaparti
e3f0e3b544
Merge branch '4.16' into main 2022-02-04 19:04:08 +05:30
davidjumani
26e4e18218
kvm: Use lscpu to get cpu max speed (#5506)
* kvm: Use lscpu to get cpu max speed

* Fix str conversion

* Reorder

* Refactor

* Apply suggestions from code review

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Updated the calling method name getCpuSpeedFromCommandLscpu

* Make it more readable

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
2022-02-04 11:37:14 +01:00
Harikrishna
f15cab16da
server: Decouple service (compute) offering and disk offering (#5008)
Currently, our compute offerings and disk offerings are tightly coupled with respect to many aspects. For example, if a compute offering is created, a corresponding disk offering entry is also created with the same ID as the reference. Also creating compute offering takes few disk-related parameters which anyway goes to the corresponding disk offering only. I think this design was initially made to address compute offering for the root volume created from a template. Also changing the offering of a volume is tightly coupled with storage tags and has to be done in different APIs either migrateVolume or resizeVolume. Changing of disk offering should be seamless and should consider new storage tags, new size and place the volume in appropriate state as defined in disk offering.

more details are mentioned here https://cwiki.apache.org/confluence/display/CLOUDSTACK/Compute+offering+and+disk+offering+refactoring

* Schema changes and disk offering column change from "type" to "compute_only"

* Few more changes

* Decoupled service offering and disk offering

* Remove diskofferingid from vminstance VO

* Decouple service offering and disk offering states

* diskoffering getsize() is only for strict disk offerings

* Fix deployVM flow

* Added new API params to compute offering creation

* Add diskofferingstrictness to serviceoffering vo under quota

* Added overrideDiskOfferingId parameter in deploy VM API which will override disk offering for the root disk both in template and ISO case

Added diskSizeStrictness parameter in create Disk offering API which will decide whether to restrict resize or disk offering change of a volume

* Fix User vm response to show proper service offering and disk offerings

* Added disk size strictness in disk offering response

* Added disk offering strictness to the service offering response

* Remove comments

* Added UI changes for Disk offering strictness in add compute offering form and Disk size strictness in add disk offering form

* Added diskoffering details to the service offering response

* Added UI changes in deployvm wizard to accept override disk offering id

* Fix delete compute offering

* Fix VM deployment from custom service offering

* Move uselocalstorage column access from service offering to disk offering

* UI: Separated compute and disk releated parameters in add compute offering wizard, also added association to disk offering

* Fixed diskoffering automatic selection on add compute offering wizard

* UI: move compute only toggle button outside the box in add compute offering wizard

* Added volumeId parameter to listDiskOfferings API and the disksizestrictness flag of the current disk offering is honored while list disk offerings

* Added configuration parameter to decide whether to check volume tags on the destination storagepool during migration

* Added disk offering change checks during resize volume operation

* Added new API changeofferingforVolume API and corresponding changes

* Add UI form for changeOfferingForVolume API

* Fix UI conflicts

* Fix service offering usage as disk offering

* Fix unit test failures

* fix user_vm_view

* Addressed review comments

* Fixed service_offering_view

* Fix service offering edit flow

* Fix service offering constructor to address custom offering

* Fix domain_router_view to get proper service offering id

* Removed unused import

* Addressed review comments and fixed update service offering flow with storage tags

* Added marvin test cases for checking disk offering strictness

* review comments addressed

* Remove system_use column from disk offering join

* update volume_view to update system_use column from service offering and not disk offering

* Fix changeOfferingForVolume API for custom disk offering

* Fix global setting implementation

* Fix list volumes, after changing system_use column from disk offering to service offering in volume_view

* Changes for override root disk offering in deployvm wizard in case of custom offering

* Fix a unit test case

* Fixed recent unit test cases with new serviceofferingvo constructor

* Fix unit test in VolumeApiServiceImpl

* Added storage id for the list disk offering API and corresponding UI changes in migrateVolume and changeOfferingForVolume flow

* Rename global configuration parameter from storage.pool.tags.disk.offering.strictness to match.storage.pool.tags.with.disk.offering

* Fix smoke test failures

* Added tool tip for migrate volume UI form

* Address review comments and fix UI form of deploy VM in case of ISO.

* Fixed resize volume UI form for data disk

* UI changes to disable override root disk size when override root disk offering is enabled

* UI fix in deploy vm wizard

* Fix listdiskoffering after rebasing with main

* Fixed UI in migrate and changeofferingfor volume to handle empty disk offering list
Removed the volume's current disk offering from listDiskOffering response list

* Added custom Iops to resize volume form and removed the current disk offering during change offering for volume UI form

* Fix false response on updateDiskOffering API

* Added search field for changeofferingforvolume UI form

* Fix resize volume and migrate volume to update volume path if DRS is applied on volume in datastore cluster

* Removed DB changes from 4.16 upgrade file

* Resolving merge conflicts with main 4.17

* Added support for auto migration and auto resize of the root volume upon changing the service offering for VM.

* UI: Added automigrate checkbox in scale VM form

* Addes since attributes to new API params

* Added shrinkOK parameter to changeofferingforvolume API

* Added shrinkOk param to UI in changeOfferingforVolume form

* Added shrinkOk flag to scaleVM and changeServiceForVirtualMachines and UI form

* Removed old foreign key constraint on IDs of service offering and disk offering

* Allow resize and automigrate of root volume if required in all cases of service offering change

* Allow only resize to higher disk size from UI

* Fixing vue syntax error

* Make UI changes to provide root disk size box when the linked disk offering is of custom

* Converted from check box to toggle in scale VM, changeoffering, resize and migrate volume forms

* Fix resize volume operation to update the VM settings

* Fix migratevolume form to pick selected storage pool id in list diskofferings API
2022-01-27 15:08:42 +05:30
Suresh Kumar Anaparti
e1b56be67c
Merge branch '4.16' into main 2022-01-11 12:46:33 +05:30
dahn
7ea2cbd889
Storage pool absent (#5841)
* simple null check

Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2022-01-11 12:18:55 +05:30
slavkap
48006de3e3
kvm: Set RAW format to RBD DATADISK (#5801)
While deploying a VM with data disk on Ceph the volume format was in
QCOW2.

Fixes #5531
2022-01-08 08:54:21 +05:30
Rohit Yadav
c84198d76d Merge remote-tracking branch 'origin/4.16'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-12-30 16:52:30 +05:30
Rohit Yadav
6728b696cd
kvm: don't always force scsi controller for aarch64 VMs (#5802)
* kvm: don't force scsi controller for aarch64 VMs

This would allow use of virtio disk controller with Ceph, etc or as
defined in the VM's root disk controller setting, rather than always
enforce SCSI.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* remove test that doesn't apply now

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* address review comment

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-12-30 15:19:32 +05:30
Rohit Yadav
9c9de74966 Merge remote-tracking branch 'origin/4.16' 2021-11-24 17:56:14 +05:30
DH Park
a85a00bed9
kvm: Fixes UEFI XML Definition Error (#5692)
UEFI bootMode 추가
2021-11-23 16:44:32 +05:30
Daniel Augusto Veronezi Salvador
b4aabadc4d
Replace string libraries with org.apache.commons.lang3.StringUtils (#5386)
* Replace google lib for lang3 and adjust methods calls

* Replace string libs by lang3

* Prohibit others string libs

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-11-18 13:41:48 +05:30
nicolas
3f79436840
Updating pom.xml version numbers for release 4.17.0.0-SNAPSHOT
Signed-off-by: nicolas <nicovazquez90@gmail.com>
2021-11-09 22:55:52 -03:00
nicolas
93c3c3b9ac
Updating pom.xml version numbers for release 4.16.1.0-SNAPSHOT
Signed-off-by: nicolas <nicovazquez90@gmail.com>
2021-11-09 22:50:22 -03:00
nicolas
44c08b5acc
Updating pom.xml version numbers for release 4.16.0.0
Signed-off-by: nicolas <nicovazquez90@gmail.com>
2021-11-04 14:14:57 -03:00
Peinthor Rene
fe1ddfb1eb
linstor-volume-plugin: Only create diskless assignments on nodes (#5661)
With diskful set to true, linstor will fail if it cannot create local
storage for the resource. Which in turn will make it impossible to have a
setup with just compute nodes on cloudstack.
2021-11-03 21:24:48 -03:00
sureshanaparti
152159b13a
kvm: Add the source disk format for disk conversion/copy using 'qemu-img convert', when specified explicitly. (#5561)
This PR adds the source disk format (in KVM) for disk conversion/copy using 'qemu-img convert', when specified explicitly

Fixes: #5516
2021-10-11 12:00:49 +05:30
davidjumani
6ac834a358
Adding AutoScaling for cks + CKS CoreOS EOL update + systemvmtemplate improvements (#4329)
Adding AutoScaling support for cks
Kubernetes PR : kubernetes/autoscaler#3629
Also replaces CoreOS with Debian
Fixes #4198

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-10-06 21:17:41 +05:30
sureshanaparti
e198edfaf8
Report the PowerFlex/ScaleIO disk copy failure during volume migration and fail the migration (#5542)
* Report the PowerFlex/ScaleIO disk copy failure during volume migration and fail the migration.

* Code improvements

* Addressed review comments
2021-10-05 14:19:52 -03:00
div8cn
52a9dbdcd2
kvm available memory calculation optimization (#5540)
* Update LibvirtComputingResource.java

* Update LibvirtComputingResource.java

* Update LibvirtComputingResource.java

re

* Update MemStat.java

* Update MemStat.java

* Update MemStatTest.java
2021-10-05 12:34:28 -03:00
Leo (Hsueh Yu-Min)
72a1c0e7f1
[KVM] Add MV Settings for virtual GPU hardware type and memory (#5513)
* KVM: Add MV Settings for virtual GPU hardware type and memory

* fix method createVideoDef argument in test package

* add available options for KVM virtual GPU hardware VM setting

* fix videoRam default value

* fix _videoRam is 0, it will use default provided by libvirt
2021-10-04 09:55:32 +05:30
Wei Zhou
09fce75047
OVS/GRE: bug fixes (#5446)
* OVS/GRE: fix failed to fetch cloudbr* info on host

* OVS/GRE: move ovstunnel.py to python3

* OVS: add test_privategw_acl_ovs_gre.py for automatic testing

(1) VR <-> VM should work
(2) Private GW should work

* OVS: fix an issue on xenserver

ovs bridges are deleted by xenserver/ovs automatically
```
[root@ref-trl-1797-x-M7-wei-zhou-xs2 ~]# grep -r xapi7 /var/log/ |grep del-br
/var/log/xensource.log:Sep 15 07:13:44 ref-trl-1797-x-M7-wei-zhou-xs2 xcp-networkd: [ info|localhost|611 |org.xen.xapi.xenops.classic events D:4a3d931cd89f|network_utils] /usr/bin/ovs-vsctl --timeout=20 -- --if-exists del-br xapi7
/var/log/daemon.log:Sep 15 07:13:45 ref-trl-1797-x-M7-wei-zhou-xs2 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=20 -- --if-exists del-br xapi7
```

which results that xe network exists but bridge does not exist, and operation stuck for 20 minutes at
```
2021-09-15 16:06:56    DEBUG [root] #### VMOPS enter  create_tunnel ####
2021-09-15 16:06:56    DEBUG [root] Creating tunnel from host 2 to host 1 with GRE key 2116
2021-09-15 16:06:56    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--timeout=0', 'wait-until', 'bridge', 'xapi7', '--', 'get', 'bridge', 'xapi7', 'name']
2021-09-15 16:26:56    DEBUG [root] bridge xapi7 for creating tunnel - VERIFIED
2021-09-15 16:26:56    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'add-port', 'xapi7', 't2116-2-1', '--', 'set', 'interface', 't2116-2-1', 'type=gre', 'options:key=2116', 'options:remote_ip=10.0.34.230']
```

* OVS/GRE: component test test_vpc_distributed_routing_offering.py passes on xenserver

* OVS: support Shared network with GRE protocol

* update #5446: use lang3.StringUtils

* OVS/GRE: VPC with distributed routing on kvm (still not working)
2021-10-03 14:47:52 +05:30
Peinthor Rene
66c39c1589
storage: Linstor volume plugin (#4994)
This adds a volume(primary) storage plugin for the Linstor SDS.
Currently it can create/delete/migrate volumes, snapshots should be possible,
but currently don't work for RAW volume types in cloudstack.

* plugin-storage-volume-linstor: notify libvirt guests about the resize
2021-09-16 10:50:58 +05:30
Wei Zhou
95ef292860
kvm: honor migrate.wait and abort vm migration job (#5388)
* kvm: honor migrate.wait and abort vm migration job

* kvm: propogate migratewait to all cloudstack agents on kvm hosts

* update #5388

* update #5388: display error msg
2021-09-15 12:38:16 -03:00
Abhishek Kumar
3d16584fa6
kvm: fix misleading error log (#5396)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-09-08 10:10:21 +05:30
Rohit Yadav
a1a3aff2b5 Merge remote-tracking branch 'origin/4.15' into main 2021-08-31 14:29:30 +05:30
slavkap
961e85eb60
Fix of creating volumes from snapshots without backup to secondary storage (#5349)
* Fix of creating volumes from snapshots without backup

When few snaphots are created onyl on primary storage, and try to create
a volume or a template from the snapshot only the first operation is
successful. Its because the snapshot is backup on secondary storage with
wrong SQL query. The problem appears on Ceph/NFS but may affects other
storage plugins.
Bypassing secondary storage is implemented only for Ceph primary storage
and it didn't cover the functionality to create volume from snapshot
which is kept only on Ceph

* Address review
2021-08-31 12:46:57 +05:30
Daniel Augusto Veronezi Salvador
159c72fa97
Externalize KVM Agent's option to change migration thread timeout (#4570)
* Externalize KVM Agent's option to change migration thread timeout

* Update javadoc

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-08-26 08:18:17 -03:00
Daniel Augusto Veronezi Salvador
83c0b61ab2
Externalize KVM Agent storage's reboot configuration (#4586)
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-08-24 01:06:00 -03:00
Daniel Augusto Veronezi Salvador
8a16729fcf
Support vm dynamic scaling with kvm (#4878)
* Create utility to centralize byte convertions

* Add/change toString definitions

* Create Libvirt handler to ScaleVmCommand

* Enable dynamic scalling VM with KVM

* Move config from interface to class and rename it

As every variable declared in interfaces are already final,
this moving will be needed to mock tests in nexts commits

* Configure VM max memory and cpu cores

The values are according to service offering or global configs

* Extract dpdk configuration to a method and test it

* Extract OS desc config to a method and test it

* Extract guest resource def to a method and test it

Improve libvirt def

* Refactor LibvirtVMDef.GuestResourceDef

* Refactor ScaleVmCommand

* Improve VMInstaVO toString()

* Refactor upgradeRunningVirtualMachine method

* Turn int variables into long on utility

* Verify if VM is scalable on KVMGuru

* Rename some KVMGuruTest's methods

* Change vm's xml to work with max memory

* Verify if service offering is dynamic before scale

* Create methods to retrieve data from domain

* Create def to hotplug memory

* Adjust the way command was scaling the VM

* Fix database persistence before executing command

* Send more info to host to improve log

* Fix var name

* Fix missing "}"

* Undo unnecessary changes

* Address review

* Fix scale validation

* Add VM prepared for dynamic scaling validation

* Refactor LibvirtScaleVmCommandWrapper and improve unit tests

* Remove duplicated method

* Add RuntimeException check

* Remove copyright from header

* Remove copyright from header

* Remove copyright from header

* Remove copyright from header

* Remove copyright from header

* Update ByteScaleUtilsTest.java

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-08-21 09:29:02 +02:00
Daniel Augusto Veronezi Salvador
65a48dcb74
Add SharedMountPoint to KVMs supported storage pool types (#4780)
* Add SharedMountPoint to KVMs supported storage pool types

* Fix live migration to iSCSI and improve logs

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-08-16 12:32:19 -03:00
Daniel Augusto Veronezi Salvador
349120f7c5
Externalize config to enable manually setting CPU topology on KVM VM (#5273)
* Externalize config to enable manually setting CPU topology on KVM VM

* Change log level

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-08-15 23:52:50 -03:00
Daniel Augusto Veronezi Salvador
82df04ecc8
Improve HA logs (#5241)
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-07-30 21:13:16 +02:00
Daniel Augusto Veronezi Salvador
7b752c3077
Externalize KVM Agent storage's timeout configuration (#5239)
* Externalize KVM Agent storage's timeout configuration

* Address @nvazquez review

* Add empty line at the end of the agent.properties file

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-07-28 15:45:27 +02:00
SadiJr
eff2da2518
Refactor and improvements for method com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVMFromSpec() (#5149)
* Refactor method createVMFromSpec

* Add unit tests

* Fix test

* Extract if block to method for add extra configs to VM Domain XML

* Split travis tests trying to isolate which test is causing an error

* Override toString() method

* Update documentation

* Fix checkstyle error (line with trailing spaces)

* Change VirtualMachineTO print of object

* Add try except to find message error. Remove after test

* Fix indent

* Trying to understanding why is happening in this code

* Refactor method createVMFromSpec

* Add unit tests

* Fix test

* Extract if block to method for add extra configs to VM Domain XML

* Split travis tests trying to isolate which test is causing an error

* Override toString() method

* Update documentation

* Fix checkstyle error (line with trailing spaces)

* Remove unnecessary comment

* Revert travis tests

Co-authored-by: SadiJr <17a0db2854@firemailbox.club>
2021-07-21 15:07:25 -03:00
dahn
6f93e5cd08
Revert "Externalize kvm agent storage timeout configuration (#4585)" (#5218)
This reverts commit 05a978c2495d1c93c798d2b845c5030ad78f6a14.
2021-07-20 09:16:43 +02:00
Daniel Augusto Veronezi Salvador
05a978c249
Externalize kvm agent storage timeout configuration (#4585)
* Externalize KVM Agent storage's timeout configuration

Created a class of constant agent's properties available to configure on "agent.properties".
Created a class to provides a facility to read the agent's properties file and get its properties.

* Refactored KVHAMonitor nested thread and changed some logs

* It has been added the timeout's config in the agent.properties file

* Rename classes

* Rename var and remove comment

* Fix typo with word "heartbeat"

* Extract multiple methods call to variables

* Add unit tests to file handler

* Increase info about the property

* Create inner class Property

* Rename method getProperty to getPropertyValue

* Remove copyright

* Remove copyright

* Extract code to createHeartBeatCommand

* Change method access from protected to private

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-07-19 09:41:50 +02:00