1162 Commits

Author SHA1 Message Date
John Bampton
e7b8326058
pre-commit auto add license for all Markdown files (#11870) 2025-10-24 16:04:22 +05:30
John Bampton
024f89dc7d
pre-commit: add oxipng a lossless PNG compression optimizer (#11065)
* pre-commit: add oxipng a lossless PNG compression optimizer

"Lossless compression is a data compression technique where the original data can be perfectly
reconstructed from the compressed data. In essence, no information is lost during compression
and decompression, making it ideal for situations where data integrity is critical"

https://en.wikipedia.org/wiki/Lossless_compression

https://github.com/oxipng/oxipng

https://github.com/oxipng/oxipng?tab=readme-ov-file#git-integration-via-pre-commit

Ran pre-commit here locally and it compressed the images on first run.

So we have less data for some images with the same quality.

Less data means less to download etc and saves bandwidth.

* Fix up from code review
2025-10-24 15:18:45 +05:30
Vishesh
d2615bb142
Add support for providing userdata to system VMs (#11654)
This PR adds support for specifying user data (cloud-init) for system VMs via Zone Scoped global settings. This allows the operators to customize the System VMs and setup monitoring, logging or execute any custom commands.

We set the user data from the global setting in /var/cache/cloud/cmdline, and use the NoCloud datasource to process user data. cloud-init service is still disabled in the system VMs and it's executed as part of the cloud-postinit service which executes the postinit.sh script.

Added global settings:
systemvm.userdata.enabled - Disabled by default. Needs to be enabled to utilize the feature.
console.proxy.vm.userdata - UUID of the User data to be used for Console Proxy
secstorage.vm.userdata - UUID of the User data to be used for Secondary Storage VM
virtual.router.userdata - UUID of the User data to be used for Virtual Routers
2025-10-08 10:44:26 +05:30
Abhisar Sinha
7dd0d6e937
add ConfigDrive to datasource_list in SharedfsVM (#11726) 2025-10-01 18:00:44 +05:30
Abhishek Kumar
2c1aad4db8
Merge remote-tracking branch 'apache/4.20' 2025-09-27 09:00:45 +05:30
Wei Zhou
5dfeb79c63
systemvm: fix failed to get script version when patch system vm or router (#10962)
This is a regression of #9900

prior to this PR
```
2025-06-05 12:28:53,992 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:) Processing command: com.cloud.agent.api.PatchSystemVmCommand
2025-06-05 12:29:25,959 DEBUG [resource.wrapper.LibvirtPatchSystemVmCommandWrapper] (AgentRequest-Handler-1:[]) (logid:) Patch result of systemVM s-368-VM: /root
2025-06-05 12:29:25,961 WARN  [resource.wrapper.LibvirtPatchSystemVmCommandWrapper] (AgentRequest-Handler-1:[]) (logid:) Failed to get the latest script version
2025-06-05 12:29:25,962 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:) Seq 2-7450361158554357406:  { Ans: , MgmtId: 32986204472275, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.PatchSystemVmAnswer":{"templateVersion":"Cloudstack Release 4.20.1 Wed May 14 05:22:13 PM UTC 2025","scriptsVersion":"5ebc6ded1a3880732363c1cdbbd54cfb
```

with this PR
```
2025-06-05 12:42:46,219 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Request:Seq 2-7450361158554357463:  { Cmd , MgmtId: 32986204472275, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.PatchSystemVmCommand":{"forced":"true","accessDetails":{"router.ip":"169.254.151.188","router.name":"s-368-VM"},"wait":"0","bypassHostMaintenance":"false"}}] }
2025-06-05 12:42:46,220 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Processing command: com.cloud.agent.api.PatchSystemVmCommand
2025-06-05 12:43:18,083 DEBUG [resource.wrapper.LibvirtPatchSystemVmCommandWrapper] (AgentRequest-Handler-2:[]) (logid:) Patch result of systemVM s-368-VM: /root
2025-06-05 12:43:18,083 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Seq 2-7450361158554357463:  { Ans: , MgmtId: 32986204472275, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.PatchSystemVmAnswer":{"templateVersion":"Cloudstack Release 4.20.1 Wed May 14 05:22:13 PM UTC 2025","scriptsVersion":"5ebc6ded1a3880732363c1cdbbd54cfb","result":"true","details":"Successfully patched systemVM s-368-VM ","wait":"0","bypassHostMaintenance":"false"}}] }
```
2025-09-27 01:36:36 +02:00
Wei Zhou
22bf8fd5d1
Merge remote-tracking branch 'apache/4.20' 2025-09-22 16:31:01 +02:00
Wei Zhou
393b5d2b77
VR: consider NICs for remote access VPN when apply dhcp entry (#11681) 2025-09-22 10:25:21 +02:00
dahn
aca8732102
[router] make a distinction between fatal errors, warnings and unknown as healthcheck result (#10710)
* [routers] distiction between fatal failure and warning or unknown on healthchecks

* UI status for router health checks

* status from scripts varied

* automation signalled errors

* revert removal of update sql

* upgradeversion

* move config item and further cleanup

* handling services better

* backwards compatible response

---------

Co-authored-by: Daan Hoogland <dahn@apache.org>
2025-09-22 11:39:05 +05:30
John Bampton
33cfedee46
pre-commit add hook check-shebang-scripts-are-executable for Shell (#10150) 2025-09-17 14:22:28 +02:00
Wei Zhou
70a4503ea1
Merge remote-tracking branch 'apache/4.20' 2025-09-11 14:04:52 +02:00
Wei Zhou
b46e29dc67
Improvement: SSL offloading with Virtual Router (#11468)
* SSL offloading with Virtual Router

* PR11468: fix pre-commit errors

* PR11468: api->getAPI/postAPI in UI

* SSL: add smoke tests for VPC in user project

* PR11468: address Daan's comments

* Fix test/integration/smoke/test_ssl_offloading.py

* SSL: remove ssl certificates when clean up account

* SSL offloading: add unit tests

* SSL offloading: UI fixes part 1

* SSL offloading: UI changes part 2

* SSL offloading: add more unit tests

* SSL offloading: more unit tests 3

* SSL offloading: wrong check

* SSL offloading: more and more unit tests

* SSL offloading: add testUpdateLoadBalancerRule5
2025-09-11 16:37:18 +05:30
Fabricio Duarte
5d32492676
IPv6 firewall: accept packets from related and established connections (#10970)
* IPv6 firewall: accept related packets and packets from established connections

* Remove rule from input chain
2025-09-10 16:37:17 +02:00
Wei Zhou
66a584624f
Update md5sum to sha512sum (#11134)
* Update md5sum to sha512sum

* Move SQL to schema-42100to42200.sql
2025-09-03 11:12:43 +02:00
Suresh Kumar Anaparti
2eb80e0361
Merge branch '4.20' 2025-08-29 15:36:16 +05:30
Suresh Kumar Anaparti
ca62a7dd50
Merge branch '4.19' into 4.20 2025-08-29 15:31:00 +05:30
Wei Zhou
05e7a257ca
VPC VR: return UNKNOWN redundant state if no guest nics (#11518)
* VPC VR: return UNKNOWN redundant state if no guest nics
2025-08-29 12:51:33 +05:30
Suresh Kumar Anaparti
1033be4b31
Updating pom.xml version numbers for release 4.22.0.0-SNAPSHOT
Signed-off-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-08-28 12:00:42 +05:30
Suresh Kumar Anaparti
f9513b47bf
Updating pom.xml version numbers for release 4.21.0.0
Signed-off-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-08-22 11:42:37 +05:30
Daan Hoogland
0b3959221b Merge branch '4.20' 2025-07-29 16:50:55 +02:00
Vishesh
f2bda46419
Upgrade noVNC from 1.4.0 to 1.6.0 (#11119) 2025-07-29 15:17:37 +02:00
Pearl Dsilva
890386e949
Allow custom NTP servers for CPVM (#11210) 2025-07-25 15:21:06 +02:00
Pearl Dsilva
0d4147f3f6
Netris Network Plugin Integration with CloudStack (#10458)
The Netris Plugin introduces Netris as a network service provider in CloudStack to be able to create and manage Virtual Private Clouds (VPCs) in CloudStack, being able to orchestrate the following network functionalities:

- Network segmentation with Netris-VXLAN isolation method
- Routing between "public" IP and network segments with an ACS ROUTED mode offering
- SourceNAT, DNAT, 1:1 NAT between "public" IP and network segments with an ACS NATTED mode offering
- Routing between VPC network segments (tiers in ACS nomenclature)
- Access Lists (ACLs) between VPC tiers and "public" network (TCP, UDP, ICMP) both as global egress rules and "public" IP specific ingress rules.
- ACLs between VPC network tiers (TCP, UDP, ICMP)
- External load balancing – between VPC network tiers and "public" IP
- Internal load balancing – between VPC network tiers
- CloudStack Virtual Router services (DHCP, DNS, UserData, Password Injection, etc…)
2025-07-25 15:26:42 +05:30
Rohit Yadav
111d87b845
console: optimise buffer sizes for faster console performance (#11221)
* console-proxy: fix stream buffer sizes to improve console performance

This bumps the input and output stream buffers to 64KiB and uses them
consistent across TLS and non-TLS based VNC connections.

This fixes #10650

Co-authored-by: Vishesh Jindal <vishesh.jindal@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* Make buffer size configurable & other improvements for CPU & memory utilisation

* Setup batching of data for TLS connections to the VNC server

* Apply suggestions from code review

* Fix buffer size for xenserver

---------

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Vishesh Jindal <vishesh.jindal@shapeblue.com>
Co-authored-by: vishesh92 <vishesh92@gmail.com>
2025-07-24 16:32:35 +05:30
John Bampton
165ab0aabe
Fix HTML license; standardize HTML code (#11067)
Co-authored-by: Vishesh <vishesh92@gmail.com>
2025-07-23 17:25:15 +02:00
Nicolas Vazquez
6adfda2818
CKS Enhancements (#9102)
CKS Enhancements:

* Ability to specify different compute or service offerings for different types of CKS cluster nodes – worker, master or etcd

* Ability to use CKS ready custom templates for CKS cluster nodes

* Add and Remove external nodes to and from a kubernetes cluster

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Update remove node timeout global setting

* CKS/NSX : Missing variables in worker nodes

* CKS: Fix ISO attach logic

* CKS: Fix ISO attach logic

* address comment

* Fix Port - Node mapping when cluster is scaled in the presence of external node(s)

* CKS: Externalize control and worker node setup wait time and installation attempts

* Fix logger

* Add missing headers and fix end of line on files

* CKS Mark Nodes for Manual Upgrade and Filter Nodes to add to CKS cluster from the same network

* Add support to deploy CKS cluster nodes on hosts dedicated to a domain

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* Support unstacked ETCD

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix CKS cluster scaling and minor UI improvement

* Reuse k8s cluster public IP for etcd nodes and rename etcd nodes

* Fix DNS resolver issue

* Update UDP active monitor to ICMP

* Add hypervisor type to CKS cluster creation to fix CKS cluster creation when External hosts added

* Fix build

* Fix logger

* Modify hypervisor param description in the create CKS cluster API

* CKS delete fails when external nodes are present

* CKS delete fails when external nodes are present

* address comment

* Improve network rules cleanup on failure adding external nodes to CKS cluster

* UI: Fix etcd template was not honoured

* UI: Fix etcd template was not honoured

* Refactor

* CKS: Exclude etcd nodes when calculating port numbers

* Fix network cleanup in case of CKS cluster failure

* Externalize retries and inverval for NSX segment deletion

* Fix CKS scaling when external node(s) present in the cluster

* CKS: Fix port numbers displayed against ETCD nodes

* Add node version details to every node of k8s cluster - as we now support manual upgrade

* Add node version details to every node of k8s cluster - as we now support manual upgrade

* update column name

* CKS: Exclude etcd nodes when calculating port numbers

* update param name

* update param

* UI: Fix CKS cluster creation templates listing for non admins

* CKS: Prevent etcd node start port number to coincide with k8s cluster start port numbers

* CKS: Set default kubernetes cluster node version to the kubernetes cluster version on upgrade

* CKS: Set default kubernetes cluster node version to the kubernetes cluster version on upgrade

* consolidate query

* Fix upgrade logic

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix CKS cluster version upgrade

* CKS: Fix etcd port numbers being skipped

* Fix CKS cluster with etcd nodes on VPC

* Move schema and upgrade for 4.20

* Fix logger

* Fix after rebasing

* Add support for using different CNI plugins with CKS

* Add support for using different CNI plugins with CKS

* remove unused import

* Add UI support and list cni config API

* necessary UI changes

* add license

* changes to support external cni

* UI changes

* Fix NPE on restarting VPC with additional public IPs

* fix merge conflict

* add asnumber to create k8s svc layer

* support cni framework to use as-numbers

* update code

* condition to ignore undefined jinja template variables

* CKS: Do not pass AS number when network ID is passed

* Fix deletion of Userdata / CNI Configuration in projects

* CKS: Add CNI configuration details to the response and UI

* Explicit events for registering cni configuration

* Add Delete cni configuration API

* Fix CKS deployment when using VPC tiers with custom ACLs

* Fix DNS list on VR

* CKS: Use Network offering of the network passed during CKS cluster creation to get the AS number

* CKS cluster with guest IP

* Fix: Use control node guest IP as join IP for external nodes addition

* Fix DNS resolver issue

* Improve etcd indexing - start from 1

* CKS: Add external node to a CKS cluster deployed with etcd node(s) successfully

* CKS: Add external node to a CKS cluster deployed with etcd node(s) successfully

* simplify logic

* Tweak setup-kube-system script for baremetal external nodes

* Consider cordoned nodes while getting ready nodes

* Fix CKS cluster scale calculations

* Set token TTL to 0 (no expire) for external etcd

* Fix missing quotes

* Fix build

* Revert PR 9133

* Add calico commands for ens35 interface

* Address review comments: plan CKS cluster deployment based on the node type

* Add qemu-guest-agent dependency for kvm based templates

* Add marvin test for CKS clusters with different offerings per node type

* Remove test tag

* Add marvin test and fix update template for cks and since annotations

* Fix marvin test for adding and removing external nodes

* Fix since version on API params

* Address review comments

* Fix unit test

* Address review comments

* UI: Make CKS public templates visible to non-admins on CKS cluster creation

* Fix linter

* Fix merge error

* Fix positional parameters on the create kubernetes ISO script and make the ETCD version optional

* fix etcd port displayed

* Further improvements to CKS  (#118)

* Multiple nics support on Ubuntu template

* Multiple nics support on Ubuntu template

* supports allocating IP to the nic when VM is added to another network - no delay

* Add option to select DNS or VR IP as resolver on VPC creation

* Add API param and UI to select option

* Add column on vpc and pass the value on the databags for CsDhcp.py to fix accordingly

* Externalize the CKS Configuration, so that end users can tweak the configuration before deploying the cluster

* Add new directory to c8 packaging for CKS config

* Remove k8s configuration from resources and make it configurable

* Revert "Remove k8s configuration from resources and make it configurable"

This reverts commit d5997033ebe4ba559e6478a64578b894f8e7d3db.

* copy conf to mgmt server and consume them from there

* Remove node from cluster

* Add missing /opt/bin directory requrired by external nodes

* Login to a specific Project view

* add indents

* Fix CKS HA clusters

* Fix build

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>

* Add missing headers

* Fix linter

* Address more review comments

* Fix unit test

* Fix scaling case for the same offering

* Revert "Login to a specific Project view"

This reverts commit 95e37563f48573780b07a038a7f48c0bc04e9b64.

* Revert "Fix CKS HA clusters" (#120)

This reverts commit 8dac16aa359faa6500ea1e1ce548169cfd08331a.

* Apply suggestions from code review about user data

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update api/src/main/java/org/apache/cloudstack/api/command/user/userdata/BaseRegisterUserDataCmd.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Refactor column names and schema path

* Fix scaling for non existing previous offering per node type

* Update node offering entry if there was an existing offering but a global service offering has been provided on scale

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-06-19 11:00:42 +05:30
Daniel Augusto Veronezi Salvador
380884ed50
Fix data being replicated on VM's metadata file in the VR (#10987) 2025-06-13 20:13:09 +02:00
Pearl Dsilva
379ee07d88 Updating pom.xml version numbers for release 4.19.4.0-SNAPSHOT
Signed-off-by: Pearl Dsilva <pearl1594@gmail.com>
2025-06-06 18:00:09 +05:30
Pearl Dsilva
b5e2c181f9 Updating pom.xml version numbers for release 4.20.2.0-SNAPSHOT
Signed-off-by: Pearl Dsilva <pearl1594@gmail.com>
2025-06-06 15:38:12 +05:30
Pearl Dsilva
c61a5eb430 Updating pom.xml version numbers for release 4.20.1.0
Signed-off-by: Pearl Dsilva <pearl1594@gmail.com>
2025-05-30 12:43:00 +05:30
Daan Hoogland
0c7d47138d Updating pom.xml version numbers for release 4.19.3.0
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-05-30 09:08:58 +02:00
Pearl Dsilva
193d6ef52f Merge branch '4.20' of https://github.com/apache/cloudstack 2025-04-18 18:29:44 -04:00
Pearl Dsilva
a95e9d7f77 Merge branch '4.19' of https://github.com/apache/cloudstack into 4.20 2025-04-18 18:29:16 -04:00
Wei Zhou
f2e688af26
VR: add bind-interfaces to /etc/dnsmasq.d/cloud.conf (#10739) 2025-04-18 13:20:13 +02:00
Daan Hoogland
4a3686297d Updating pom.xml version numbers for release 4.19.3.0-SNAPSHOT
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-02-25 10:43:11 +01:00
Daan Hoogland
24b7c66251 Merge branch '4.20' 2025-02-24 14:33:12 +01:00
Daan Hoogland
4e321d4356 Updating pom.xml version numbers for release 4.19.2.0
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-02-20 09:32:07 +01:00
Wei Zhou
789f94b664
VR: fix duplicated lines in .htaccess (#10254) 2025-02-14 08:57:48 -05:00
Daan Hoogland
0dcb8da03a Merge branch '4.20' 2025-02-12 16:54:05 +01:00
Daan Hoogland
4f3e8e8c5a Merge branch '4.19' into 4.20 2025-02-12 15:00:51 +01:00
Vishesh
ae1d7cc860
Fix private gateway acl on static routes (#10262) 2025-02-10 18:50:30 +01:00
dahn
d453c63848
cleanup VM IP after expunge in redundant VPC (#10183)
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
2025-02-09 09:34:20 +01:00
Wei Zhou
90c960eeed
VPC VR: fix ACL between tier and private gateway (#10268) 2025-02-04 16:00:58 +01:00
Daan Hoogland
bd874eaa44 Merge release branch 4.20 to main
* 4.20:
  systemvm: fix keystore is reset when patch a systemvm (#9900)
  no retrieval of null hosts (#10175)
  upgrade: consider multiple hypervisors and secondary storages (#10046)
  CheckOnHostCommand: add missing timeout setting (#9677)
2025-01-13 11:23:51 +01:00
Daan Hoogland
e2cfddb1b6 Merge branch '4.19' into 4.20 2025-01-13 11:23:14 +01:00
Wei Zhou
35fe19f096
systemvm: fix keystore is reset when patch a systemvm (#9900) 2025-01-13 09:40:12 +01:00
John Bampton
0f3ea8ceb2
Add pre-commit hook fix-byte-order-marker (#10147)
https://github.com/pre-commit/pre-commit-hooks?tab=readme-ov-file#fix-byte-order-marker
2025-01-03 09:40:43 +01:00
Daan Hoogland
2daffa34f2 Merge release branch 4.20 to main
* 4.20:
  VR: fix site-2-site VPN if split connections is enabled (#10067)
  UI: fix cannot open 'Edit tags' modal for static routes (#10065)
  Update ownership selection component to be language independent (#10052)
  Support to enable/disable VM High Availability manager and related alerts (#10118)
2024-12-30 13:35:30 +01:00
Daan Hoogland
41c27e116c Merge release branch 4.19 to 4.20
* 4.19:
  VR: fix site-2-site VPN if split connections is enabled (#10067)
  UI: fix cannot open 'Edit tags' modal for static routes (#10065)
2024-12-30 13:07:56 +01:00
Wei Zhou
5c01cff304
VR: fix site-2-site VPN if split connections is enabled (#10067) 2024-12-28 12:09:36 +01:00