* cloudstack: add support for EL10
This adds support for Fedora 40 and (upcoming) EL10 distro to be used
as mgmt/usage server, mysql/nfs & KVM host. Python3 version has changed
to 3.12.9 which isn't automatically determining the python-path.
* python: WIP code, this fails right now
Need to discuss/check if we can skip this code. Where/how is cgroup
setup used with KVM agent.
* prep cloudutils to be EL10 ready
Fixes issue for Fedora, it was running old EL6 hooks which isn't
applicable for modern Fedora version that are closer to EL8/9/10
This PR allows securing the console access through CloudStack to the virtual machines running on KVM. The secure access is achieved through the generated certificates for the CA Framework in CloudStack, that provides mutual TLS connections between agents. These certificates are used to also secure the connection between the console proxies and the VNC ports for VM console access.
This feature is only supported on the KVM hypervisor
Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Secure+KVM+VNC+connection+using+the+CA+framework
This masks sockets such as virtqemud.socket virtqemud-ro.socket
virtqemud-admin.socket, and services virtqemud virtnetworkd virtstoraged
that block libvirtd to be used a monolithic service. Useful for both EL9
and existing Ubuntu 22.04, and newer distros.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
This PR adds support for SUSE Leap 15.2+ as a KVM hypervisor and Management / Usage server on Cloudstack
Related docs link apache/cloudstack-documentation#224
Co-authored-by: Michael <35783820+mib1185@users.noreply.github.com>
Inclusivity changes for CloudStack
- Change default git branch name from 'master' to 'main' (post renaming/changing default git branch to 'main' in git repo)
- Rename some offensive words/terms as appropriate for inclusiveness.
This PR updates the default git branch to 'main', as part of #4887.
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* DB : Add support for MySQL 8
- Splits commands to create user and grant access on database, the old
statement is no longer supported by MySQL 8.x
- `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove
that from db.properties conn parameters
For mysql-server 8.x setup the following changes were added/tested to
make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and
then restart the mysql-server process:
server_id = 1
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=1000
log-bin=mysql-bin
binlog-format = 'ROW'
default-authentication-plugin=mysql_native_password
Notice the last line above, this is to reset the old password based
authentication used by MySQL 5.x.
Developers can set empty password as follows:
> sudo mysql -u root
ALTER USER 'root'@'localhost' IDENTIFIED BY '';
In libvirt repository, there are two related commits
2019-08-23 13:13 Daniel P. Berrangé ● rpm: don't enable socket activation in upgrade if --listen present
2019-08-22 14:52 Daniel P. Berrangé ● remote: forbid the --listen arg when systemd socket activation
In libvirt.spec.in
/bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* ubuntu16: fix unable to add host if cloudbrX is not configured
while add a ubuntu16.04 host with native eth0 (cloudbrX is not configured),
the operation failed and I got the following error in /var/log/cloudstack/agent/setup.log
```
DEBUG:root:execute:ifconfig eth0
DEBUG:root:[Errno 2] No such file or directory
File "/usr/lib/python2.7/dist-packages/cloudutils/serviceConfig.py", line 38, in configration
result = self.config()
File "/usr/lib/python2.7/dist-packages/cloudutils/serviceConfig.py", line 211, in config
super(networkConfigUbuntu, self).cfgNetwork()
File "/usr/lib/python2.7/dist-packages/cloudutils/serviceConfig.py", line 108, in cfgNetwork
device = self.netcfg.getDefaultNetwork()
File "/usr/lib/python2.7/dist-packages/cloudutils/networkConfig.py", line 53, in getDefaultNetwork
pdi = networkConfig.getDevInfo(dev)
File "/usr/lib/python2.7/dist-packages/cloudutils/networkConfig.py", line 157, in getDevInfo
elif networkConfig.isBridge(dev) or networkConfig.isOvsBridge(dev):
```
The issue is caused by commit 9c7cd8c2485412bc847b2c2473b962fa01435b24
2017-09-19 16:45 Sigert Goeminne ● CLOUDSTACK-10081: CloudUtils getDevInfo function will now return "bridge" instead o
* ubuntu16: Stop service libvirt-bin.socket while add a host
service libvirt-bin.socket will be started when add a ubuntu 16.04 host
DEBUG:root:execute:sudo /usr/sbin/service libvirt-bin start
However, libvirt-bin service will be broken by it after restarting
Stopping service libvirt-bin.socket will fix the issue.
An example is given as below.
```
root@node32:~# /etc/init.d/libvirt-bin restart
[ ok ] Restarting libvirt-bin (via systemctl): libvirt-bin.service.
root@node32:~# virsh list
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
root@node32:~# systemctl stop libvirt-bin.socket
root@node32:~# /etc/init.d/libvirt-bin restart
[ ok ] Restarting libvirt-bin (via systemctl): libvirt-bin.service.
root@node32:~# virsh list
Id Name State
----------------------------------------------------
```
* ubuntu16: Diable libvirt default network
By default, libvirt will create default network virbr0 on kvm hypervisors.
If vm uses the same ip range 192.168.122.0/24, there will be some issues.
In some cases, if we run tcpdump inside vm, we will see the ip of kvm hypervisor as source ip.
* task/add distribution detection, remove export CGROUP_DAEMON='cpu:/virt'
* taks/add exit values
* change exit codes
* split redhat and centos / change ubuntu to /etc/lsb-release file
* turn around centos and rhel, we need first check for centos then rhel
* set variable distro to None
This extends securing of KVM hosts to securing of libvirt on KVM
host as well for TLS enabled live VM migration. To simplify implementation
securing of host implies that both host and libvirtd processes are
secured with management server's CA plugin issued certificates.
Based on whether keystore and certificates files are available at
/etc/cloudstack/agent, the KVM agent determines whether to use TLS or
TCP based uris for live VM migration. It is also enforced that a secured
host will allow live VM migration to/from other secured host, and an
unsecured hosts will allow live VM migration to/from other unsecured
host only.
Post upgrade the KVM agent on startup will expose its security state
(secured detail is sent as true or false) to the managements server that
gets saved in host_details for the host. This host detail can be accesed
via the listHosts response, and in the UI unsecured KVM hosts will show
up with the host state of ‘unsecured’. Further, a button has been added
that allows admins to provision/renew certificates to KVM hosts and can
be used to secure any unsecured KVM host.
The `cloudstack-setup-agent` was modified to accept a new flag `-s`
which will reconfigure libvirtd with following settings:
listen_tcp=0
listen_tls=1
tcp_port="16509"
tls_port="16514"
auth_tcp="none"
auth_tls="none"
key_file = "/etc/pki/libvirt/private/serverkey.pem"
cert_file = "/etc/pki/libvirt/servercert.pem"
ca_file = "/etc/pki/CA/cacert.pem"
For a connected KVM host agent, when the certificate are
renewed/provisioned a background task is scheduled that waits until all
of the agent tasks finish after which libvirt process is restarted and
finally the agent is restarted via AgentShell.
There are no API or DB changes.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This includes test related fixes and code review fixes based on
reviews from @rafaelweingartner, @marcaurele, @wido and @DaanHoogland.
This also includes VMware disk-resize limitation bug fix based on comments
from @sateesh-chodapuneedi and @priyankparihar.
This also includes the final changes to systemvmtemplate and fixes to
code based on issues found via test failures.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This fixes test failures around VMware with the new systemvmtemplate.
In addition:
- Does not skip rVR related test cases for VMware
- Removes rc.local
- Processes unprocessed cmd_line.json
- Fixed NPEs around VMware tests/code
- On VMware, use udevadm to reconfigure nic/mac address than rebooting
- Fix proper acpi shutdown script for faster systemvm shutdowns
- Give at least 256MB of swap for VRs to avoid OOM on VMware
- Fixes smoke tests for environment related failures
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Migrate to embedded Jetty server.
- Improve ServerDaemon implementation.
- Introduce a new server.properties file for easier configuration.
- Have a single /etc/default/cloudstack-management to configure env.
- Reduce shaded jar file, removing unnecessary dependencies.
- Upgrade to Spring 5.x, upgrade several jar dependencies.
- Does not shade and include mysql-connector, used from classpath instead.
- Upgrade and use bountcastle as a separate un-shaded jar dependency.
- Remove tomcat related configuration and files.
- Have both embedded UI assets in uber jar and separate webapp directory.
- Refactor systemd and init scripts, cleanup packaging.
- Made cloudstack-setup-databases faster, using `urandom`.
- Remove unmaintained distro packagings.
- Moves creation and usage of server keystore in CA manager, this
deprecates the need to create/store cloud.jks in conf folder and
the db.cloud.keyStorePassphrase in db.properties file. This also
remove the need of the --keystore-passphrase in the
cloudstack-setup-encryption script.
- GZip contents dynamically in embedded Jetty
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
MySQLdb has been deprecated and is also not supported in Python 3.
mysql.connector is a connector written in Python which talks the
native MySQL protocol without any external code.
https://dev.mysql.com/doc/connector-python/en/
Add support for not (re)starting server after cloud-setup-management.This adds an option to the cloud-setup-management script to not start the management server after a successful configuration of it.
The primary motivation for this is to avoid circular dependency issues on systems that use systemd. When calling cloud-setup-management from a unit with a Before= directive on a service depending on
cloudstack-management, the process will deadlock because /usr/bin/service will delegate to systemd, which is waiting for the Before service to start.
Executing the cloud-setup-management script with this new `--no-start` option will simply leave the management server stopped after a successful configuration. systemd can then be bypassed with `export _SYSTEMCTL_SKIP_REDIRECT=1` and using the init.d script.
* pr/1162:
Add support for not (re)starting server after cloud-setup-management.
Signed-off-by: Remi Bergsma <github@remi.nl>
This adds an option to the cloud-setup-management script to not start
the management server after a successful configuration of it. The
primary motivation for this is to avoid circular dependency issues on
systems that use systemd. When calling cloud-setup-management from a
unit with a Before= directive on a service depending on
cloudstack-management, the process will deadlock because
/usr/bin/service will delegate to systemd, which is waiting for the
Before service to start.
CLOUDSTACK-9049: fix Centos7 with Tomcat7 packaging and python libsCLOUDSTACK-9049: Fix Centos7 with Tomcat7 packaging and python libs
* pr/1052:
CLOUDSTACK-9049: fix Centos7 with Tomcat7 packaging and python libs * adjust library to support tomcat7 config files * adjust centos7 spec to use tomcat7 config files from deployment * add option to use tomcat7 files in management server setup
reorder content to match original
Signed-off-by: Remi Bergsma <github@remi.nl>
* adjust library to support tomcat7 config files
* adjust centos7 spec to use tomcat7 config files from deployment
* add option to use tomcat7 files in management server setup
CLOUDSTACK-9039: Fix paths for logging Ubuntu ManagementFix paths for logging Ubuntu Management.
How to test:
Install via DEB packages...
* pr/1039:
CLOUDSTACK-9039: Fix paths for logging Ubuntu Management.
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9029: Proper support to identify CentOS 7 version numberhttps://issues.apache.org/jira/browse/CLOUDSTACK-9029
* pr/1033:
CLOUDSTACK-9029: Proper support to identify CentOS 7 version number
Signed-off-by: Remi Bergsma <github@remi.nl>
If a custom resource (kvm/libvirt implementation) is defined in agent.properties
don't override with the default, but check and fallback to the default if
resource property not defined
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 536a8b22c8865dc94281bce6267930a63e03ab77)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
