35115 Commits

Author SHA1 Message Date
Pearl Dsilva
830f3061bc
SystemVM optimizations (#5831)
* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* refactor patching logic - certs

* move logic of fixing rootdiskcontroller from upgrade to kubernetes service

* add livepatch option to restart network & vpc

* smooth upgrade of cks clusters

* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* add cgroup config for containerd

* add systemd config for kubelet

* add additional info during image registry config

* address comments

* add temp links of download.cloudstack.org

* address part of the comments

* address comments

* update containerd config - as version has upgraded to 1.5 from 1.4.12 in 4.17.0

* address comments - simplify

* fix vue3 related icon changes

* allow network commands when router template version is lower but is patched

* add internal LB to the list of routers to be patched on network restart with live patch

* add unit tests for API param validations and new helper utilities - file scp & checksum validations

* perform patching only for non-user i.e., system VMs

* add test to validate params

* remove unused import

* add column to domain_router to display software version and support networkrestart with livePatch from router view

* Requires upgrade column to consider package (cloud-scripts) checksum to identify if true/false

* use router software version instead of checksum

* show N/A if no software version reported i.e., in upgraded envs

* fix deb failure

* update pom to official links of systemVM template
2022-04-21 13:40:19 -03:00
DK101010
18d9c6dfc6
fix mismatching between db uuids and custom attributes uuids (#5382)
* fix mismatching between db uuids and custom attributes uuids

during the datastore cluster creation, cloudstack could not
recognize the existing primary storage and create a new one because
uuid format not equal

* remove method call setUuid

* add upgrade step to fix faulty pool uuids

* adapt method to transform uuid each time

* extract error msg

* rm unused import

* add exception to log error as parameter

* adapt sql to fetch wrong uuids

* rm spaces

* move upgrade code to Upgrade41610to41700

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
2022-04-21 10:37:40 -03:00
Nicolas Vazquez
5cf8064467
Fix: Prevent NPE on disk offering search while listing VMs (#6287) 2022-04-20 11:11:40 -03:00
Wei Zhou
56fb7ac128
UI: hide Virtual Routers tab for domain admins (#6289) 2022-04-20 11:00:20 -03:00
Pearl Dsilva
25f92c50f0
ui: Fix Internal LB LB rule column and missing translation (#6288) 2022-04-20 10:33:43 -03:00
Wei Zhou
9f076ff10d
UI: checksum field is optional for direct-download templates on kvm (#6290) 2022-04-20 10:28:12 -03:00
Rohit Yadav
40193a04eb .github: fix coverage issue, don't kick pkging automatically
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-20 16:37:32 +05:30
DK101010
ccac1a383f
Feat/add vdisk UUID to list volume (#5848)
* get vdisk uuid from vcenter and store it into database

* add vdisk uuid as external_uuid to listVolume response

* add sql upgrade file

* Update vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* update sql add column external_uuid

* Update server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java

Co-authored-by: Wei Zhou <weizhou@apache.org>

* adapt param description for externalUuid

* add 'idempotent column add' to create external_uuid col

* rename method to getExternalDiskUUID

* remove line disk_offering.system_use

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
2022-04-19 23:34:09 -03:00
Pearl Dsilva
c339f6eef9
ui: Fix bulk deletion of ssh key pairs (#6286) 2022-04-19 22:47:59 -03:00
SadiJr
4313c3def7
Allow users to view reserved System VM IPs, if they're already allocated to user (#5902)
* Allow users to view reserved system VM IPs, if this IPs are already allocated to any user VM

* Fix checkstyle

* Address reviews

* Address reviews

* Apply @weizhouapache changes

Credits to @weizhouapache, and my sincere thanks for the help.

Co-authored-by: SadiJr <sadi@scclouds.com.br>
Co-authored-by: SadiJr <17a0db2854@firemailbox.club>
2022-04-19 23:15:15 +02:00
nvazquez
7ea0dea479
Merge branch '4.16' 2022-04-19 17:11:56 -03:00
Suresh Kumar Anaparti
b2338f7158
Updated reset configuration, to return the updated config value in the response (#6284) 2022-04-19 17:07:51 -03:00
Wei Zhou
a5014a28a6
New feature: give access permission of networks to other accounts in same domain (#5769)
* Enhancement: create Shared networks and VPC private gateways by users

* UI bug fix: pass correct domainid in CreateSharedNetworkForm

* Update #5730: fix test failure with test_guest_vlan_range.py

* Update #5730: fix test failure with test_persistent_network.py

* Update #5730: Add since to new API commands and API parameters

* Update #5730: Get first physical network for VPC private gateway if other ways do not work

* Update #5730: code optimization (return !offering.isSpecifyVlan())

* Update #5730: fix hard-coded network offering id in test_pvlan.py

* Update #5730: skip access check on the network owner if the owner is ROOT/system

* Update #5730: overlap check on cidr/startip/endip

* Update #5730: add methods to get accountid/domainid of shared networks

* Update #5730: improve integration tests

* Update #5730: update as per GutoVeronezi's comments

* Network Sharing: give network access permission to other accounts within a domain

* network: update ip in lb/pf/dnat tables when update vm nic ip

* Update #5757: create 3 separated methods for DNAT/LB/PF update

* travis: install python3-setuptools

* Network Sharing: update integration test

* Update #5769: Remove NetworkPermission.Ops

* Update #5769: Update as per Daan's comments

* Update #5769: Update as per Suresh's comments

* Update #5769: fix UI bug that accounts/projects are not listed

* Update #5769: fix domain admin can deploy vm on L2 network of other users

* Update #5769: Remove method listPermittedNetworkIdsByDomains in NetworkPermissionDao

* Update #5769: Skip network operation permissions check for root admin

* UI: fix create Isolated/L2 network form

* Update #5730: fix create Shared network form

* Update #5769: fix domain admin can deploy vm on L2 network of other users

* test: fix test_storage_policy.py

* Update #5769: fix remove_nic in test_network_permissions.py

* Update #5769: extract some codes to a method

* Update #5769: fix add/remove nic by domain admin

* Update #5769: allow domain admin to enable/disable static nat and create port forwarding rules

* Update #5769: update integration test

* Update #5769: fix unit test AssignLoadBalancerTest.java

* Update #5769: allow normal users to share network permission to other users on UI

* Update #5769: fix small UI bug with label

* Update #5769: Support L2 network as associated network

* test: sleep 30s after restarting mgt server in test_kubernetes_supported_versions.py to fix test failures with test_secondary_storage.py

* Update #5784: revert part of changes in #2420

* Update #5757: invert if condition to reduce code indentation

* Update #5769: fix regular user cannot create L2 network

* Update #5769: Add associated nework id and name in private gateway response

* Update #5769: list networks by networkfilter=Account on UI

* Update #5769: fix ui issue when list private gateways or create shared network if no isolated networks

* Update #5769: fix vue ui warnings

* Update #5679: add BaseResponseWithAssociatedNetwork and extract method setResponseAssociatedNetworkInformation

* Update #5679: extract some methods in VpcManagerImpl.java

* Update #5679: Update smoke tests as per Daan's comments

* Update #5769: fix vpc with private gateways cannot be removed when remove an acount

* Update #5769: fix unit test failures after merging latest main

* Update #5769: fix schema-41610to41700.sql

* Update #5769: fix Request failed due to empty network offering list on UI

* Update #5769: Throw exception when account is not found by name

* Update #5769: display a warning message if network offering list is empty

* Update #5769: fix an UI bug caused by previous commit b286cb76774e301d15cf2b3b6ea316eaa306018f

* Update #5769: fix UI bugs due to vue3 merge

* Update #5769: fix issue due to account type refactoring

* Update #5769: fix ui bugs due to vue3

* Update #5769: fix issue due to vue3 upgrade

* Update #5769: fix issue due to vue3 upgrade part 2

* Update #5769: fix issue due to vue3 upgrade part 3

* Update #5769: highlight default scope when create shared network on UI

* Update #5769: fix domain list is not loaded on UI

* Update #5769: fix restart/delete shared network by normal users

* Update #5769: fix restart domain-scope shared network by domain admin

* Update #5769: fix 3 UI bugs (1) double networks in list; (2) icon of first items in list; (3) account/project autoselect

* Update #5769: fix 2 ui bugs; (1) selected project is not changed when change domain; (2) no network should be selected by default

* Update #5769: fix update shared networks by domain admin/regular user

* Update #5769: fix Flicking warning message about the empty network offerings

* Update #5769: display associated network name in shared network info card

* Update #5769: fix create private gateway form

* Update #5769: fix network lists in project view

* Update #5769: fix duplicated networks in network dropdown

* Update #5769: fix failed to create shared network if associated L2 network is Setup

* Update #5769: check AccessType.OperateEntry on network in its implementation

* Revert "Update #5769: check AccessType.OperateEntry on network in its implementation"

This reverts commit c42c489e5bb26e2c780ac1be79bc8ca299a23f6e.

* Update #5769: fix keyword search in list guest vlans
2022-04-19 11:29:31 -03:00
nvazquez
334891a4b9
Merge branch '4.16' 2022-04-19 09:49:21 -03:00
Nicolas Vazquez
365966dd0a
UI: Fix custom unconstrained for a zone does not show CPU speed (#6285) 2022-04-19 09:48:26 -03:00
Pearl Dsilva
8a1b1250e8
ui: remove mandatory rule on root disk controller field while registering / updating a template (#6279) 2022-04-19 08:49:38 -03:00
Leo
70122007bb
Updated SAML2 auth sessionkey cookie path (#6149)
This change will set the sessionkey under the /client/api path. This commit should prevent duplicate sessionkey cookies from being set on both /client (incorrect) and /client/api (correct). Prior to this commit, the /client version was being set while the /client/api version remained unchanged with an invalid sessionkey. As a result, subsequent requests after the SAML2 authentication would immediately fail with an invalid session and results in the user being logged out.

The sessionkey is now set explicitly for the /client/api path which should fix this issue, regardless of the SSO URL and path that's being used.
2022-04-18 17:16:20 +05:30
Rohit Yadav
15928f8cc8 .github: detect and kick packaging jobs
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-18 17:04:44 +05:30
Rohit Yadav
cc139464f3 .github: use event specific sha and base_sha for comparing changes
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-18 15:31:20 +05:30
Rohit Yadav
1b2461b200 .github: debug github event using env variable
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-18 15:24:48 +05:30
Rohit Yadav
b2086595ca .github: debug github event
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-18 15:19:13 +05:30
Pearl Dsilva
14dd467a43
ui: Incorrect column key specified in secondary store column filter (#6275) 2022-04-18 15:07:10 +05:30
Rohit Yadav
976464ccb9 .github: enable fork-point while finding list of changes files
Hit same issue as https://github.com/tj-actions/changed-files/issues/355

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-18 14:46:32 +05:30
Rohit Yadav
023305e86d .github: fetch all history for comparing PR changes
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-18 14:37:11 +05:30
Rohit Yadav
dcb9143330 .github: use pull_request_target which has access to token
This tries to add support for forked repo pull requests, to be able
to run quality checks on it and send comments on the PR. Hopefully,
this solves the issues faced recently.

Due to the way pull_request_target works, I can't send this as a PR
so committing directly to `main` and then sending a dummy PR from a fork
to test if it works as documented by github and elsewhere.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-18 14:28:05 +05:30
John Bampton
c265df965e
Fix spelling (#6185)
* Fix spelling

* Update services/secondary-storage/server/src/main/java/org/apache/cloudstack/storage/resource/NfsSecondaryStorageResource.java

Co-authored-by: SadiJr <sadijacinto@gmail.com>

Co-authored-by: Rohit Yadav <rohityadav89@gmail.com>
Co-authored-by: SadiJr <sadijacinto@gmail.com>
2022-04-16 00:55:11 +05:30
Rohit Yadav
3ac91aa452 pom: add sonar project properties
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-15 20:44:01 +05:30
Rohit Yadav
2dcdc56115
.github: run coverage on pull request (#6265)
The pull_request_target doesn't seem to work for PRs from forks,
this reverts to use and kick the workflow for pull_request instead.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-15 20:35:22 +05:30
John Bampton
cfa4f33940
.github/linters: Enable flake8 W293 blank line contains whitespace (#6268)
https://www.flake8rules.com/rules/W293.html
2022-04-15 20:32:52 +05:30
Gabriel Beims Bräscher
50b2dc2789
server: Fix #6263 Cannot scale VM with custom offering (#6267)
* When scaling with custom offering, which changes only CPU/Memory and keeps same disk offering an exception is thrown.

This commit fixes such cases by checking if the operation is happening on a custom service offering.

* Improve the unit tests that cover null objects.
2022-04-15 20:28:31 +05:30
Hoang Nguyen
fecc5254de
ui: fixes style ui issues (#6261)
This PR for fixes some UI Bugs

 The project selected color is not displayed after switching project
 Project view, the cards are not separated but stick together
 Dashboard: The color of the selected zone is not displayed
 Dark mode: Hover through row tables that are discolored
2022-04-15 20:27:16 +05:30
Wei Zhou
aa0197c141
kvm: truncate vnc password to 8 chars (#6244)
This PR truncates the vnc password of kvm vms to 8 chars to support latest versions of libvirt.
2022-04-15 20:26:42 +05:30
slavkap
4004dfcfd8
StorPool storage plugin (#6007)
* StorPool storage plugin

Adds volume storage plugin for StorPool SDS

* Added support for alternative endpoint

Added option to switch to alternative endpoint for SP primary storage

* renamed all classes from Storpool to StorPool

* Address review

* removed unnecessary else

* Removed check about the storage provider

We don't need this check, we'll get if the snapshot is on StorPool be
its name from path

* Check that current plugin supports all functionality before upgrade CS

* Smoke tests for StorPool plug-in

* Fixed conflicts

* Fixed conflicts and added missed Apache license header

* Removed whitespaces in smoke tests

* Added StorPool plugin jar for Debian

the StorPool jar will be included into cloudstack-agent package for
Debian/Ubuntu
2022-04-14 11:12:01 -03:00
Rohit Yadav
9067938a0d .github: try kicking coverage via pull_request_target sync
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-14 18:03:36 +05:30
Rohit Yadav
cd24cf2daa
.github: improve coverage run (#6238)
Add support to detect and kick UI builds and run quality
build and report coverage for the PR.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-14 17:51:36 +05:30
Pearl Dsilva
05c907b3a8
ui: Allow editing VM and template settings (#6262) 2022-04-14 10:37:30 +02:00
nvazquez
a1fb44e41f
Merge branch '4.16' 2022-04-14 01:22:09 -03:00
Pearl Dsilva
08bfb4444a
ui: Add project switch to the Kubernetes tab (#6260) 2022-04-14 01:21:05 -03:00
Pearl Dsilva
b4ac98efa3
ui: Display action buttons in Project Accounts Tab view if project Admin (#6257) 2022-04-14 01:20:11 -03:00
Wei Zhou
bbb4ffa593
UI: fix dedicate public ip range to domain (#6258) 2022-04-14 01:19:23 -03:00
nvazquez
3674aff4d4
Merge branch '4.16' 2022-04-14 01:17:38 -03:00
Rakesh
c7941278f7
Allow creating snapshot from VM snapshot (#4739)
If `kvm.snapshot.enabled` is set to false then we cant create snapshot from
VM snapshot. With this change, its possible to create snapshot from VM
snapshot even when the global setting is set to false.
Note that you still cant directly create a snapshot from volume though
2022-04-14 01:16:59 -03:00
Nicolas Vazquez
fbf77978e1
Fix: Allow disabling the login attempts mechanism for disabling users (#6254)
* Fix: Allow disabling the login attempts mechanism for disabling users

* Refactor
2022-04-14 01:11:43 -03:00
Rohit Yadav
fc31b527b1
maven: upgrade to latest spring-framework release (#6250)
Upgrades spring framework and reload4j dependencies

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-13 21:53:12 +05:30
dahn
6140903004
local versions of .env ignored (#6256)
Co-authored-by: Daan Hoogland <dahn@onecht.net>
2022-04-13 09:46:07 -03:00
nvazquez
65dc2df896
Merge branch '4.16' 2022-04-13 08:45:48 -03:00
slavkap
42a92dcdd3
Extract the IO_URING configuration into the agent.properties (#6253)
When using advanced virtualization the IO Driver is not supported. The
admin will decide if want to enable/disable this configuration from
agent.properties file. The default value is true
2022-04-13 08:43:35 -03:00
nvazquez
67002b67ec
Merge branch '4.16' 2022-04-13 07:22:56 -03:00
Wei Zhou
91a5f0e285
server: honor global setting system.vm.default.hypervisor as first option when deploy VRs (#6160) 2022-04-13 07:22:22 -03:00
Wei Zhou
daa13af626
UI: fix icon user-delete-outlined for release dedicated public ip range (#6255) 2022-04-13 09:20:56 +02:00