35927 Commits

Author SHA1 Message Date
Wei Zhou
c8d6e50539
VMware: add support for 8.0b (8.0.0.2), 8.0c (8.0.0.3) (#7380)
* VMware: add support for 8.0b (8.0.0.2)

* VMware 8: add new guest os mappings in VirtualMachineGuestOsIdentifier

The full list can be found at https://developer.vmware.com/apis/1355/vsphere

* VMware: get guest os mappings of parent version

* VMware8: remove guest os mappings for 8.0.0.2

* VMware8: fix code smells

* vmware: remove annotations in VmwareVmImplementerTest which caused 0.0% code coverage

* VMware8: add a unit test case

* VMware: add support for 8.0c (8.0.0.3)

* VMware8: move to CloudStackVersion.getVMwareParentVersion

* VMware: add support for 8.0u1 (8.0.1.0)

* Copy engine/schema/src/main/java/com/cloud/upgrade/GuestOsMapper.java from PR 6979

* Copy engine/schema/src/main/java/com/cloud/storage/dao/GuestOSHypervisorDao.java from PR 6979

* VMware: ignore the last number in VMware versions

* VMware: copy guest os mapping from 8.0 to 8.0.1

* VMware: add unit tests in VmwareVmImplementerTest.java

* Copy engine/schema/src/test/java/com/cloud/upgrade/GuestOsMapperTest.java from PR 6979

* VMware8: retry vm poweron if fails due to exception "File system specific implementation of Ioctl[file] failed"

This fixes a weird issue on vmware8. When power on a vm, sometimes it fails due to error

2023-04-27 07:04:43,207 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-442:ctx-cdd42b03 10.0.32.133, job-105/job-106, cmd: StartCommand) (logid:8a24a607) StartCommand failed due to [Exception: java.lang.RuntimeException
Message: File system specific implementation of Ioctl[file] failed
].
java.lang.RuntimeException: File system specific implementation of Ioctl[file] failed
        at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:426)
        at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.powerOn(VirtualMachineMO.java:288)

in vmware.log on ESXi host, it shows

2023-04-27T09:20:41.713Z In(05)+ vmx - Power on failure messages: File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to lock the file
2023-04-27T09:20:41.713Z In(05)+ vmx - Cannot open the disk '/vmfs/volumes/7b29c876-ac102328/i-2-167-VM/ROOT-167.vmdk' or one of the snapshot disks it depends on.
2023-04-27T09:20:41.713Z In(05)+ vmx - Module 'Disk' power on failed.
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to start the virtual machine.

There is a KB article for it, but I still do not know why and how to fix it.
https://kb.vmware.com/s/article/1004232

* VMware: extract to method powerOnVM

* vmware: fix mistake in logs

* vmware8: use curl instead of wget to fix test failures

Traceback (most recent call last):
  File "/root/test_internal_lb.py", line 555, in test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
    self.execute_internallb_roundrobin_tests(vpc_offering)
  File "/root/test_internal_lb.py", line 641, in execute_internallb_roundrobin_tests
    client_vm, applb.sourceipaddress, max_http_requests)
  File "/root/test_internal_lb.py", line 497, in run_ssh_test_accross_hosts
    (e, clienthost.public_ip))
AssertionError: list index out of range: SSH failed for VM with IP Address: 10.0.52.187

and

sshClient: DEBUG: {Cmd: /usr/bin/wget -T3 -qO- --user=admin --password=password http://10.1.2.253:8081/admin?stats via Host: 10.0.52.188} {returns: ["/usr/bin/wget: '/usr/lib/libpcre.so.1' is not an ELF file", "/usr/bin/wget: can't load library 'libpcre.so.1'"]}

* VMware: correct guest OS names in hypervisor mappings for VMware 8.0

el9 and variants were introduced by https://github.com/apache/cloudstack/pull/7059
they are supported with guest os identifiers since VMware 8.0

see https://vdc-repo.vmware.com/vmwb-repository/dcr-public/c476b64b-c93c-4b21-9d76-be14da0148f9/04ca12ad-59b9-4e1c-8232-fd3d4276e52c/SDK/vsphere-ws/docs/ReferenceGuide/vim.vm.GuestOsDescriptor.GuestOsIdentifier.html

* VMware: add Ubuntu 20.04 and 22.04 support for vmware 7.0+

* PR7380: only add guest os mappings for Ubuntu 20.04

* PR7380: Correct RHEL9 guest os names and others for VMware 8.0

* PR7380: correct guest os names on 8.0.0.1 as well

* PR7380: remove Windows 12 and Windows Server 2025 which are not released yet
2023-08-17 10:42:42 +02:00
Charles Queiroz
e8b491177f
server: Replace Hashtable with LinkedHashMap in createIsoResponse (#7844)
* Replace Hashtable with LinkedHashMap in createIsoResponse

This change replaces the use of Hashtable with LinkedHashMap in the `createIsoResponse` method of `ViewResponseHelper`.
 The reason for this modification is to maintain the insertion order of entries, which isn't the case with Hashtable.
  This could lead to more predictable results and behaviors in calling methods.

* Replace Hashtable with LinkedHashMap in view response creation methods

Changed Hashtable to LinkedHashMap in various response creation methods within ViewResponseHelper class.
This modification ensures an ordered iteration which is beneficial for scenarios where the insertion order of responses needs to be maintained consistently.

---------

Co-authored-by: Sina Kashipazha <soreana@users.noreply.github.com>
2023-08-16 08:58:41 +02:00
dahn
cf249f64a3
api: Force-stop descriptions (#7866) 2023-08-16 08:31:32 +02:00
dahn
19aacad46b
UI: Show iso urls (#7869)
* add url to details

* some cleanup
2023-08-15 13:51:53 +02:00
Wei Zhou
aa02d9b3c1
test: skip live storage migration on CentOS 7 (#7862)
since #7570, The detail 'Host.OS' of centos7 host is changed from 'CentOS' to 'CentOS Linux'
2023-08-15 08:39:18 +02:00
Alexandru Bagu
a47a4f4ad4
CPVM: fix cpu usage for console vm when using vnc over websockets (#6970)
Co-authored-by: Alexandru Bagu <alexandru.bagu@hotmail.com>
2023-08-14 15:45:04 +02:00
Fabricio Duarte
72e3491cef
server: Fix allocation of more public IPs than the account's limit (#7832) 2023-08-14 10:33:29 +02:00
Fabricio Duarte
2c6072273b
VMware: Fix casting exception during backup NIC synchronization (#7831) 2023-08-14 10:20:33 +02:00
Wei Zhou
bf9c71430b
CKS: check access when get kubernetescluster config (#7854) 2023-08-14 10:02:20 +02:00
Rohit Yadav
feb9509547
ui: assorted improvements (#7833)
This PR aims to polish the UI with following tweaks and changes:

- Increase resource and os-logo icons both in list view, user-menu bar and VM deployment form
- Fix css issues in VM deployment form when resource icons are on some of the templates/isos but not all
- Replace edit icon in the resource icon editting button on the infocard, in resource view
- Fix css marging/padding issue for nav bar and left-branding/logo
- Introduce a new Limits option in the user menu, to allow users to see their own limits when they log in
- Rename resource tab to limits tab for accounts, project and domains
- Introduce a new copy-label component, that can be clicked to copy strings; use in info-card and list view for entites such as IP addresses and UUIDs
- Add router-link to /zones/ in case of user-accounts (when /zone isn't routable in the UI)
- Show better list of nics and ssh keys pairs in infocard for VM resource view
- Standardise most resources to show state/status columns right after resource name (wherever applicable)
- Remove displayname column in VM list view, add cpu number and memory by default
- Add k8s version column in k8s list view
- Add size and phy size columns in case of template and ISOs list view, only for root/domain admins
- Add phy network router-link in case of guest VLAN list view; rearrange columns list for consistency
- Add snapshot phy size column in the snapshot list view; and router-link for volume in the snapshot list view; and missing/useful details in the volume snapshot details view
- Add a create and add data disk feature in Instances tab, just like we've add nic feature in the same

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-11 14:00:04 +05:30
Rohit Yadav
29c7b3167e
cloudstack: bump to use 4.18.1.0 systemvmtemplate (#7828)
* cloudstack: bump to use 4.18.1.0 systemvmtemplate

Use new 4.18.1.0 systemvmtemplate

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* fix cd ISO url

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

---------

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-11 07:38:47 +02:00
Wei Zhou
d9319e645c
storpool: fix pom.xml which caused docker image build failures (#7845)
steps to reproduce the issue:
- git clone https://github.com/apache/cloudstack.git
- cd cloudstack
- rm -rf .git/
- run `mvn -P developer,systemvm clean install`

Without this PR, it fails with error

```
 > [ 8/10] RUN mvn -Pdeveloper -Dsimulator -DskipTests clean install:
668.1 [ERROR] Failed to execute goal pl.project13.maven:git-commit-id-plugin:4.9.10:revision (get-the-git-infos) on project cloud-plugin-storage-volume-storpool: .git directory is not found! Please specify a valid [dotGitDirectory] in your pom.xml -> [Help 1]
```
2023-08-10 16:28:49 +02:00
Wei Zhou
1605b2f0db
server: remove registered userdata when cleanup an account (#7777) 2023-08-10 14:33:59 +02:00
João Jandre
fdb23dae40
server: Use max secondary storage defined on the account during upload (#7441) 2023-08-10 11:39:40 +02:00
Wei Zhou
b32ba953cf
test: upgrade kubernetes versions to 1.25.0/1.26.0 (#7685)
* test: upgrade versions to 1.25.0/1.26.0

* PR7685: move settings to test_data.py
2023-08-10 09:48:08 +02:00
fermosan
fa58f59619
kvm: Added VNI Devices as normal bridge slave devs (#7836)
This will allow for VXLAN configurations to utilize tags on the physical network of a zone
2023-08-10 08:59:28 +02:00
Wei Zhou
a1af03f413
noVNC: fix JP keyboard on vmware7+ which uses websocket URL (#7694)
* noVNC: fix JP keyboard on vmware7+ which uses websocket URL

* noVNC: cleanup rfb.js

* noVNC: fix < and > on JP keyboard

* noVNC: fix Caps lock on JP keyboard
2023-08-09 11:43:16 +02:00
Sina Kashipazha
9df23f951b
Prometheus exporter fix cpu/memory usage labels (#7629) 2023-08-07 20:47:03 +02:00
dahn
dea1373480
server: clean network offerings for domain on remove (#7775) 2023-08-07 09:56:58 +02:00
Rohit Yadav
4f85c5568f
engine/schema: fix CPU cap limitation for upgraded environment (#7808)
CPU cap limitation was enabled as part of
https://github.com/apache/cloudstack/pull/6420 that changes behaviour
for existing environments. The CPU cap limitation on KVM causes
systemvms to not start or be really slow in nested and virtualised
environments.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-07 10:19:33 +05:30
mprokopchuk
ab0297ea9b
VM.CREATE/VOLUME.DELETE/VOLUME.DESTROY not being emitted (#7760)
VM.CREATE/VOLUME.DELETE/VOLUME.DESTROY not being emitted

* Update server/src/main/java/com/cloud/vm/UserVmManagerImpl.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* Update api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

---------

Co-authored-by: Maxim Prokopchuk <mprokopchuk@apple.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-08-07 10:18:17 +05:30
Rohit Yadav
dc5e4f3ec6
Allow KVM overcommit to work without reducing minimum VM memory when vm ballooning is disabled (#7810)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
2023-08-06 10:39:14 +02:00
dahn
6b21825746
make details optional for addGuestOs API (#7812) 2023-08-04 08:44:29 +02:00
kishankavala
57779a8586
Remove nic allocation txn during Vm deploy. During Vm deploy failure, this will allow cleanup of any nics that are successfully provisioned (#7809) 2023-08-04 08:42:38 +02:00
dahn
b4032d9984
include all VMs when projects selected not just for admins (#7667) 2023-08-03 16:06:11 +02:00
dahn
c86684fbdd
UI: allow new keys for VM details (#7793) 2023-08-03 04:58:22 +02:00
slavkap
da7fc72c66
Refactoring StorPool's smoke tests (#7392)
* Removed the hardcoded StorPool endpoint from tests

- removed the hardcoded enpoint of StorPool primary storage from tests
- added the git commit information into the maven build

* Convert indents to spaces

* update git-commit-id-plugin version
2023-08-02 14:37:37 +08:00
Wei Zhou
87f218c68c
UI: decode userdata in EditVM dialog (#7796) 2023-08-01 09:10:26 +02:00
dahn
60ab494c25
packaging: unalias cp before package upgrade (#7722) 2023-08-01 01:34:05 +02:00
dahn
2bb079189c
make NoopDbUpgrade do a systemvm template check (#7564) 2023-07-31 16:08:14 +02:00
dahn
fd13184dda
UI unit test: fix expected values (#7792) 2023-07-31 12:46:22 +02:00
dahn
a2eb10304f
UI: Filter templates by zone and hypervisor type when reinstall a VM (#7739) 2023-07-31 07:30:24 +02:00
dahn
d127d7939d
KVM: fix SSVM starting when overprovisioning memory (#7663) 2023-07-28 11:23:30 +02:00
Wei Zhou
f4a4417e4c
pom.xml: add property project.systemvm.template.location (#7706)
* pom.xml: add property project.systemvm.template.location

* pom.xml: update project.systemvm.template.location
2023-07-28 11:05:44 +02:00
Wei Zhou
d89d40595c
cloudutils: fix adding rocky9 host failure due to missing /etc/sysconfig/libvirtd (#7779) 2023-07-28 10:27:44 +02:00
dahn
d958882282
server: get id from persisted object ReservationVO (#7785) 2023-07-28 05:16:33 +02:00
dahn
01e5c25523
search in (too) large result sets (#7766)
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
2023-07-27 15:03:12 +02:00
Wei Zhou
cdf01879ec
ui: fix 404 error when list volumes of system vms (#7772)
This PR fixes #7662
2023-07-26 15:59:45 +05:30
Wei Zhou
3f2449f3d8
packaging: install tzdata-java on centos7/centos8 (#7768)
When install cloudstack on latest rocky8 server with java-11-openjdk-11.0.20.0.8-2.el8.x86_64, management server cannot be started due to error
```
Caused by: java.lang.Error: java.io.FileNotFoundException: /usr/lib/jvm/java-11-openjdk-11.0.20.0.8-2.el8.x86_64/lib/tzdb.dat (No such file or directory)
```

It is because the package `tzdata-java` is not automatically installed.
(It was installed with java-11-openjdk-11.0.18.0.10-2.el8_7.x86_64, but not with latest java-11-openjdk-11.0.20.0.8-2.el8.x86_64)
2023-07-26 03:20:53 +02:00
dahn
f9b7bcfd10
add remove tag to remove acl item method (#7750) 2023-07-25 10:55:44 +02:00
Wei Zhou
4000fd09f1
UI: fix userdataid or userdata content is passed when userdata append/override is unchecked (#7758) 2023-07-25 10:01:53 +02:00
Rohit Yadav
bde80f14aa
Fix NPE in management server logs due to /proc/cpuinfo output (#7765)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-07-25 04:43:33 +02:00
Wei Zhou
6dd2ce86b5
UI: add step to create bearer token for kubernetes dashboard (#7764)
Since Kubernetes v1.24.0, there is no auto-generation of secret-based service account token due to security reason. see https://github.com/kubernetes/kubernetes/pull/108309

To access kubernetes dashboard, users need to create a service account and an optional long-lived Bearer Token for the service account.
2023-07-25 03:32:58 +02:00
Marcus Sorensen
63216425d5
Set encrypted PowerFlex disk format correctly (#7735)
Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-07-24 13:13:46 +05:30
Wei Zhou
90baae3dcd
utils: fix RBD URI if credentials contains slash (#7708) 2023-07-24 08:29:01 +02:00
Abhishek Kumar
f057f4b412
ui: fix userdata base64 encoding (#7749)
* ui: fix userdata abse64 encoding

Fixes #7748

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-07-21 12:38:08 +02:00
Harikrishna
80ca3acf15
Allow encrypted volume migration for PowerFlex volumes (#7757) 2023-07-21 10:08:21 +03:00
Abhishek Kumar
cee7a713aa
server: clear resource reservation and increment resource count in a transaction (#7724)
This PR addresses rare case of potential overlap of resource reservation and resource count.
For different resource types there could be some delay between incrementing of the resource count and clearing of the earlier done reservation. This may result in failures when there are parallel deployments happening.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-07-21 10:55:51 +05:30
Wei Zhou
56d98ea2e7
SSVM: 'allow from' private IP in other SSVMs if the public IP is in allowed internal sites cidrs (#7288)
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-07-20 13:38:05 +02:00
dahn
9c5e489d82
eof added to StorPoolStatsCollector (#7754) 2023-07-19 12:36:44 +02:00