2352 Commits

Author SHA1 Message Date
Vishesh
3b11663d87
Fix failure on agent reconnection (#8089) 2023-10-26 16:54:36 +02:00
Abhishek Kumar
a2ec1f3777
marvin,test: fix directdownload template checksum test (#8096)
* marvin,test: fix directdownload template checksum test

During failure while deploying a VM with wrong checksum template, VM may be left in Error state. This PR adds code to delete such VM.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove unnecessary logs

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-10-19 19:40:03 +02:00
Abhishek Kumar
540c7b802f
test: add test for standalone snapshot (#8104)
Fixes #8034

Adds the following test for a backed-up snapshot (original template and VM deleted beforehand):
- Create volume from snapshot
- Create a template from the snapshot and deploy a VM using it

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-10-19 09:48:16 +05:30
Abhishek Kumar
065abe2a3b
test,refactor: fix test_project_resources cleanup (#8097)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-10-16 16:24:59 +02:00
Harikrishna
a9f3af85cb
Default value of force should be false for template delete operation (#7731)
* default value of force should be false

* Added force flag in tests
2023-10-10 15:49:57 +05:30
Wei Zhou
246bb24b0f Updating pom.xml version numbers for release 4.18.2.0-SNAPSHOT
Signed-off-by: Wei Zhou <weizhou@apache.org>
2023-09-12 17:26:53 +02:00
Wei Zhou
4bdff06acd Updating pom.xml version numbers for release 4.18.1.0
Signed-off-by: Wei Zhou <weizhou@apache.org>
2023-09-07 08:50:50 +02:00
Wei Zhou
8dc5fdd067
server: fix cannot get systemvm ips in dedicated ranges (#7144)
This fixes #6698
2023-08-25 11:36:39 +02:00
Marcus Sorensen
a0702279aa
server Don't allow inadvertent deletion of hidden details via API (#7880)
* Don't allow inadvertent deletion of hidden details via API

* Update VM details unit test ensuring system/hidden details not removed

* Update test/integration/component/test_update_vm.py

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-08-24 10:45:39 +02:00
Wei Zhou
78bdde9e98
AutoScaling: support Managed User Data (#7769) 2023-08-22 11:07:16 +02:00
Wei Zhou
c8d6e50539
VMware: add support for 8.0b (8.0.0.2), 8.0c (8.0.0.3) (#7380)
* VMware: add support for 8.0b (8.0.0.2)

* VMware 8: add new guest os mappings in VirtualMachineGuestOsIdentifier

The full list can be found at https://developer.vmware.com/apis/1355/vsphere

* VMware: get guest os mappings of parent version

* VMware8: remove guest os mappings for 8.0.0.2

* VMware8: fix code smells

* vmware: remove annotations in VmwareVmImplementerTest which caused 0.0% code coverage

* VMware8: add a unit test case

* VMware: add support for 8.0c (8.0.0.3)

* VMware8: move to CloudStackVersion.getVMwareParentVersion

* VMware: add support for 8.0u1 (8.0.1.0)

* Copy engine/schema/src/main/java/com/cloud/upgrade/GuestOsMapper.java from PR 6979

* Copy engine/schema/src/main/java/com/cloud/storage/dao/GuestOSHypervisorDao.java from PR 6979

* VMware: ignore the last number in VMware versions

* VMware: copy guest os mapping from 8.0 to 8.0.1

* VMware: add unit tests in VmwareVmImplementerTest.java

* Copy engine/schema/src/test/java/com/cloud/upgrade/GuestOsMapperTest.java from PR 6979

* VMware8: retry vm poweron if fails due to exception "File system specific implementation of Ioctl[file] failed"

This fixes a weird issue on vmware8. When power on a vm, sometimes it fails due to error

2023-04-27 07:04:43,207 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-442:ctx-cdd42b03 10.0.32.133, job-105/job-106, cmd: StartCommand) (logid:8a24a607) StartCommand failed due to [Exception: java.lang.RuntimeException
Message: File system specific implementation of Ioctl[file] failed
].
java.lang.RuntimeException: File system specific implementation of Ioctl[file] failed
        at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:426)
        at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.powerOn(VirtualMachineMO.java:288)

in vmware.log on ESXi host, it shows

2023-04-27T09:20:41.713Z In(05)+ vmx - Power on failure messages: File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to lock the file
2023-04-27T09:20:41.713Z In(05)+ vmx - Cannot open the disk '/vmfs/volumes/7b29c876-ac102328/i-2-167-VM/ROOT-167.vmdk' or one of the snapshot disks it depends on.
2023-04-27T09:20:41.713Z In(05)+ vmx - Module 'Disk' power on failed.
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to start the virtual machine.

There is a KB article for it, but I still do not know why and how to fix it.
https://kb.vmware.com/s/article/1004232

* VMware: extract to method powerOnVM

* vmware: fix mistake in logs

* vmware8: use curl instead of wget to fix test failures

Traceback (most recent call last):
  File "/root/test_internal_lb.py", line 555, in test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
    self.execute_internallb_roundrobin_tests(vpc_offering)
  File "/root/test_internal_lb.py", line 641, in execute_internallb_roundrobin_tests
    client_vm, applb.sourceipaddress, max_http_requests)
  File "/root/test_internal_lb.py", line 497, in run_ssh_test_accross_hosts
    (e, clienthost.public_ip))
AssertionError: list index out of range: SSH failed for VM with IP Address: 10.0.52.187

and

sshClient: DEBUG: {Cmd: /usr/bin/wget -T3 -qO- --user=admin --password=password http://10.1.2.253:8081/admin?stats via Host: 10.0.52.188} {returns: ["/usr/bin/wget: '/usr/lib/libpcre.so.1' is not an ELF file", "/usr/bin/wget: can't load library 'libpcre.so.1'"]}

* VMware: correct guest OS names in hypervisor mappings for VMware 8.0

el9 and variants were introduced by https://github.com/apache/cloudstack/pull/7059
they are supported with guest os identifiers since VMware 8.0

see https://vdc-repo.vmware.com/vmwb-repository/dcr-public/c476b64b-c93c-4b21-9d76-be14da0148f9/04ca12ad-59b9-4e1c-8232-fd3d4276e52c/SDK/vsphere-ws/docs/ReferenceGuide/vim.vm.GuestOsDescriptor.GuestOsIdentifier.html

* VMware: add Ubuntu 20.04 and 22.04 support for vmware 7.0+

* PR7380: only add guest os mappings for Ubuntu 20.04

* PR7380: Correct RHEL9 guest os names and others for VMware 8.0

* PR7380: correct guest os names on 8.0.0.1 as well

* PR7380: remove Windows 12 and Windows Server 2025 which are not released yet
2023-08-17 10:42:42 +02:00
Wei Zhou
aa02d9b3c1
test: skip live storage migration on CentOS 7 (#7862)
since #7570, The detail 'Host.OS' of centos7 host is changed from 'CentOS' to 'CentOS Linux'
2023-08-15 08:39:18 +02:00
Wei Zhou
b32ba953cf
test: upgrade kubernetes versions to 1.25.0/1.26.0 (#7685)
* test: upgrade versions to 1.25.0/1.26.0

* PR7685: move settings to test_data.py
2023-08-10 09:48:08 +02:00
mprokopchuk
ab0297ea9b
VM.CREATE/VOLUME.DELETE/VOLUME.DESTROY not being emitted (#7760)
VM.CREATE/VOLUME.DELETE/VOLUME.DESTROY not being emitted

* Update server/src/main/java/com/cloud/vm/UserVmManagerImpl.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* Update api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

---------

Co-authored-by: Maxim Prokopchuk <mprokopchuk@apple.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-08-07 10:18:17 +05:30
slavkap
da7fc72c66
Refactoring StorPool's smoke tests (#7392)
* Removed the hardcoded StorPool endpoint from tests

- removed the hardcoded enpoint of StorPool primary storage from tests
- added the git commit information into the maven build

* Convert indents to spaces

* update git-commit-id-plugin version
2023-08-02 14:37:37 +08:00
Vishesh
594c70dde0
Sync precommit config from main (#7732)
Co-authored-by: John Bampton <jbampton@users.noreply.github.com>
Co-authored-by: dahn <daan@onecht.net>
2023-07-07 11:18:16 +02:00
Wei Zhou
973b0e28fd test: fix Super-Linter Check error in storpool tests
2023-06-27 08:42:08 [INFO]   File:[/github/workspace/test/integration/plugins/storpool/TestEncryptedVolumes.py]
2023-06-27 08:42:08 [ERROR]   Found errors in [flake8] linter!
2023-06-27 08:42:08 [ERROR]   Error code: 1. Command output:
------
/github/workspace/test/integration/plugins/storpool/TestEncryptedVolumes.py:681:113: W292 no newline at end of file
------
2023-06-27 08:42:08 [INFO]   ---------------------------
2023-06-27 08:42:08 [INFO]   File:[/github/workspace/test/integration/plugins/storpool/sp_util.py]
2023-06-27 08:42:08 [ERROR]   Found errors in [flake8] linter!
2023-06-27 08:42:08 [ERROR]   Error code: 1. Command output:
------
/github/workspace/test/integration/plugins/storpool/sp_util.py:798:31: W292 no newline at end of file
------
2023-06-27 10:47:25 +02:00
slavkap
faaf72b1a4
Volume encryption support for StorPool plug-in (#7539)
Supported Virtual machine operations:
- live migration of VM to another host
- virtual machine snapshots (group snapshot without memory)
- revert VM snapshot
- delete VM snapshot
Supported Volume operations:
- attach/detach volume
- live migrate volume between two StorPool primary storages
- volume snapshot
- delete snapshot
- revert snapshot
2023-06-26 11:24:51 +02:00
Harikrishna
40cc10a73d
Allow volume migrations in ScaleIO within and across ScaleIO storage clusters (#7408)
* Live storage migration of volume in scaleIO within same storage scaleio cluster

* Added migrate command

* Recent changes of migration across clusters

* Fixed uuid

* recent changes

* Pivot changes

* working blockcopy api in libvirt

* Checking block copy status

* Formatting code

* Fixed failures

* code refactoring and some changes

* Removed unused methods

* removed unused imports

* Unit tests to check if volume belongs to same or different storage scaleio cluster

* Unit tests for volume livemigration in ScaleIOPrimaryDataStoreDriver

* Fixed offline volume migration case and allowed encrypted volume migration

* Added more integration tests

* Support for migration of encrypted volumes across different scaleio clusters

* Fix UI notifications for migrate volume

* Data volume offline migration: save encryption details to destination volume entry

* Offline storage migration for scaleio encrypted volumes

* Allow multiple Volumes to be migrated with migrateVirtualMachineWithVolume API

* Removed unused unittests

* Removed duplicate keys in migrate volume vue file

* Fix Unit tests

* Add volume secrets if does not exists during volume migrations. secrets are getting cleared on package upgrades.

* Fix secret UUID for encrypted volume migration

* Added a null check for secret before removing

* Added more unit tests

* Fixed passphrase check

* Add image options to the encypted volume conversion
2023-06-21 11:57:05 +05:30
Abhishek Kumar
b030ea19a4 Merge remote-tracking branch 'apache/4.17' into 4.18 2023-06-06 13:12:12 +05:30
Abhishek Kumar
f298008215 Merge remote-tracking branch 'apache/4.16' into 4.17 2023-06-06 13:11:24 +05:30
Pearl Dsilva
bd5d478a3a
Refactor test and change IP range (#6512)
* Refactor test and change IP range

* countdown from 254 to allow multiple pseudo public nets counting up from 0

* cleanup

* location of asserts improved

---------

Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
2023-06-05 16:39:33 +02:00
Abhishek Kumar
7319debc87
cks: k8s cluster on vpc tier (#7479)
* cks: cluster on vpc tier

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* wip

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* python fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix trailing space

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-06-01 15:29:06 +05:30
dahn
897dad619b
marvin: replace encodestring for encodebytes (#7027)
Fixes: #6962
2023-05-08 12:51:12 +05:30
Abhishek Kumar
330a330555
test: add smoke test for user role for userdata crud api (#7332)
* test: add smoke test for user role for userdata crud api

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address review comment

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-03-31 15:16:29 +05:30
Daan Hoogland
05cda2729f Updating pom.xml version numbers for release 4.18.1.0-SNAPSHOT
Signed-off-by: Daan Hoogland <daan@onecht.net>
2023-03-15 19:38:14 +01:00
Daan Hoogland
0574087284 Updating pom.xml version numbers for release 4.18.0.0
Signed-off-by: Daan Hoogland <daan@onecht.net>
2023-03-11 09:35:41 +01:00
dahn
597a8030b7
integration test fixes for 4.18 (#7190)
* cleanup in resource housekeeping

* cleanup in resource housekeeping

* Update test/integration/smoke/test_deploy_vm_extra_config_data.py

Co-authored-by: dahn <daan.hoogland@gmail.com>

* revert most cleanup code

* Revert "cleanup in resource housekeeping"

This reverts commit 1d140bd215b29c7a5ec4cbfb9eabf0465a8864d9.

* add extra config to VirtualMachine.create(..)

---------

Co-authored-by: Wei Zhou <weizhou@apache.org>
2023-02-14 08:20:11 +01:00
Harikrishna
a3feccf70c
User two factor authentication (#6924)
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-02-13 09:14:17 +01:00
Wei Zhou
f3e8f39f2b
smoke test: fix test_vm_deployment_planner (#7183) 2023-02-09 15:19:57 +01:00
Rahul Agarwal
0efa4f6e51
Make displayText in createNetwork optional (#7112)
Co-authored-by: Rahul Agarwal <rahul.agarwal@shapeblue.com>
Co-authored-by: Stephan Krug <stekrug@icloud.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-02-08 14:41:59 +01:00
Wei Zhou
62e342c1bc
utils,framework/db: Introduce new database encryption cipher based on AesGcmJce (#7003) 2023-02-02 16:25:49 +01:00
David Jumani
c774b865c9
Tungsten integration (#7065)
Co-authored-by: rtodirica <rtodirica@ena.com>
Co-authored-by: Huy Le <huylm@unitech.vn>
Co-authored-by: radu-todirica <Radu.Todirica@ness.com>
Co-authored-by: Huy Le <minh.le@ext.ewerk.com>
Co-authored-by: Simon Weller <siweller77@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-02-01 09:19:53 +01:00
Suresh Kumar Anaparti
d8c7e34b38
Improve global settings UI to be more intuitive/logical (#5797)
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Co-authored-by: nvazquez <nicovazquez90@gmail.com>
Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-01-31 11:23:43 +01:00
Abhishek Kumar
3b6ce97097
infra: edge zones (#6840)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-01-31 09:36:45 +01:00
nvazquez
83c2bfacd8
Merge branch '4.17' 2023-01-30 07:53:58 -03:00
Abhishek Kumar
028ca74fb6
ui,server,api: resource metrics improvements (#6803)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-01-30 09:48:03 +01:00
Nicolas Vazquez
c78a777d3a
Fix: memory leak on volume allocation (#7136) 2023-01-30 09:44:50 +01:00
Nicolas Vazquez
eac357cb77
kvm: Secure KVM VNC Console Access Using the CA Framework (#7015)
This PR allows securing the console access through CloudStack to the virtual machines running on KVM. The secure access is achieved through the generated certificates for the CA Framework in CloudStack, that provides mutual TLS connections between agents. These certificates are used to also secure the connection between the console proxies and the VNC ports for VM console access.

This feature is only supported on the KVM hypervisor

Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Secure+KVM+VNC+connection+using+the+CA+framework
2023-01-27 17:22:06 +05:30
John Bampton
d74f64a2e1
Use lowercase HTTP header field names so we are compatible with HTTP/2 (#7006) 2023-01-23 11:17:54 +01:00
John Bampton
52c321a0c6
Fix spelling (#7087) 2023-01-16 10:56:07 +01:00
dahn
08d54da938
tests: assess prerequisite before doing the actual test (#7040)
This PR skips the tagging tests that fail due an ISO not uploading (which is not part of the functionality being tested)

relates to #7021
2023-01-12 15:21:37 +05:30
John Bampton
00426ede46
Add pre-commit workflow with 3 Git hooks (#6273)
Co-authored-by: dahn <daan@onecht.net>
2023-01-11 12:25:31 +01:00
dahn
df96af3de4
delete F5 and SRX plugins (#7023) 2023-01-11 12:07:44 +01:00
slavkap
b392084950
Enable live volume migration for StorPool and small fixes (#6661) 2023-01-10 18:21:39 +01:00
Sina Kashipazha
20306d6129
Allow creating atmost 1 physical network with null tag (#6781) 2023-01-05 21:36:31 +01:00
Wei Zhou
1380c604b1
server: add Host Control Plane State to uservm and systemvm response (#6946)
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-01-05 09:59:28 +01:00
Pearl Dsilva
3044d63a8b
Configurable MTU for VR (#6426)
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-01-04 09:42:24 +01:00
dahn
73a610da94
revert cleanup of test_acl_sharenetworks (#7033) 2022-12-31 14:45:03 +01:00
Rohit Yadav
fab4fc2a14 Merge remote-tracking branch 'origin/4.17' 2022-12-22 22:55:15 +05:30