* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency
* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp
* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup
* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes
* Add ssh to k8s nodes details in the Access tab on the UI
* test
* Refactor ca/cert patching logic
* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script
* remove all references of systemvm.iso
* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs
* fix script timeout
* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand
* remove commented code + change core user to cloud for cks nodes
* Update ownership of ssh directory
* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)
* Add UI changes + move changes from patch file to runcmd
* test: validate performance for template modification during seeding
* create vms folder in cloudstack-commons directory - debian rules
* remove logic for on the fly template convert + update k8s test
* fix syntax issue - causing issue with shared network tests
* Code cleanup
* refactor patching logic - certs
* move logic of fixing rootdiskcontroller from upgrade to kubernetes service
* add livepatch option to restart network & vpc
* smooth upgrade of cks clusters
* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency
* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp
* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup
* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes
* Add ssh to k8s nodes details in the Access tab on the UI
* test
* Refactor ca/cert patching logic
* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script
* remove all references of systemvm.iso
* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs
* fix script timeout
* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand
* remove commented code + change core user to cloud for cks nodes
* Update ownership of ssh directory
* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)
* Add UI changes + move changes from patch file to runcmd
* test: validate performance for template modification during seeding
* create vms folder in cloudstack-commons directory - debian rules
* remove logic for on the fly template convert + update k8s test
* fix syntax issue - causing issue with shared network tests
* Code cleanup
* add cgroup config for containerd
* add systemd config for kubelet
* add additional info during image registry config
* address comments
* add temp links of download.cloudstack.org
* address part of the comments
* address comments
* update containerd config - as version has upgraded to 1.5 from 1.4.12 in 4.17.0
* address comments - simplify
* fix vue3 related icon changes
* allow network commands when router template version is lower but is patched
* add internal LB to the list of routers to be patched on network restart with live patch
* add unit tests for API param validations and new helper utilities - file scp & checksum validations
* perform patching only for non-user i.e., system VMs
* add test to validate params
* remove unused import
* add column to domain_router to display software version and support networkrestart with livePatch from router view
* Requires upgrade column to consider package (cloud-scripts) checksum to identify if true/false
* use router software version instead of checksum
* show N/A if no software version reported i.e., in upgraded envs
* fix deb failure
* update pom to official links of systemVM template
* StorPool storage plugin
Adds volume storage plugin for StorPool SDS
* Added support for alternative endpoint
Added option to switch to alternative endpoint for SP primary storage
* renamed all classes from Storpool to StorPool
* Address review
* removed unnecessary else
* Removed check about the storage provider
We don't need this check, we'll get if the snapshot is on StorPool be
its name from path
* Check that current plugin supports all functionality before upgrade CS
* Smoke tests for StorPool plug-in
* Fixed conflicts
* Fixed conflicts and added missed Apache license header
* Removed whitespaces in smoke tests
* Added StorPool plugin jar for Debian
the StorPool jar will be included into cloudstack-agent package for
Debian/Ubuntu
* Create profiles to download systemvm-templates
* Rename profiles
* Add support to pass necessary flags to the packaging jobs
* Escape flags
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
* packaging: display First Install and Onboarding Message
* Update #5851: Update as per Rohit's comments
* Update #5851: display package name in help message
* Update #5851: display links of installed cloudstack version on UI
* Update #5851: fix vue warnings
Support for centos7 and centos8
Prior PR #2915 causes the cloudstack-usage daemon to not be able to find the mysql-ha jar file.
This happens because the /etc/default/cloudstack-usage file points to the old location for the mysql-ha jar file, and thus is unable to load it. This prior PR installs this jar into a more common area with other jar files, and thus is not really a desired location for the cloudstack-usage daemon to get it from, as it will cause cloudstack-usage to load other plugins that it doesn't need.
Fixes: #3974
Updates the DB properties (with strict / full property string search) having master and slave(s), with source and replica(s) respectively on upgrade (for inclusiveness).
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
This PR prepares marvin and tests for python3. it was part of #4479, until nose2 was decided to be abandoned from that PR.
Re-PR of #4543 and #3730 to enable cooperation
Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Gabriel Beims Bräscher <gabriel@apache.org>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This changes deb and rpm packaging to build the UI using npm and bundle
it in the `cloudstack-management` package and a new `cloudstack-ui`
package. The `cloudstack-ui` package will install the UI under
`/usr/share/cloudstack-ui/`. For both packages the config.json will not
be overridden on upgrade and hosted at /etc/cloudstack/management
for the cloudstack-mangement package, and at /etc/cloudstack/ui for the
cloudstack-ui package. The cloudstack-ui package is for advanced users
who only want the UI want to setup reverse proxy (separate hosting of UI).
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* DB : Add support for MySQL 8
- Splits commands to create user and grant access on database, the old
statement is no longer supported by MySQL 8.x
- `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove
that from db.properties conn parameters
For mysql-server 8.x setup the following changes were added/tested to
make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and
then restart the mysql-server process:
server_id = 1
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=1000
log-bin=mysql-bin
binlog-format = 'ROW'
default-authentication-plugin=mysql_native_password
Notice the last line above, this is to reset the old password based
authentication used by MySQL 5.x.
Developers can set empty password as follows:
> sudo mysql -u root
ALTER USER 'root'@'localhost' IDENTIFIED BY '';
In libvirt repository, there are two related commits
2019-08-23 13:13 Daniel P. Berrangé ● rpm: don't enable socket activation in upgrade if --listen present
2019-08-22 14:52 Daniel P. Berrangé ● remote: forbid the --listen arg when systemd socket activation
In libvirt.spec.in
/bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Currently CloudStack is using logging frameworks as log4j and Java util logging, logging wrappers as slf4j and Apache common logging.
Here changes are to made it uniform, using only log4j framework.
Removed Java util logging, slf4j and Apache common logging.
This adds support for JDK11 in CloudStack 4.14+:
- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Extract systemvm.iso using bsdtar if available.
Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
* New dependency for CentOS 7 and Debian: bsdtar
bsdtar can extract iso images without mounting.
Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
* Remove all 'mount' and 'umount' command call(s).
Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
Use iproute to get default network interface
This uses `ip route` than `route` without an explicit path (/bin, /sbin
etc.) to query the default network interface on the system.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* kvm/security_group: Make Security Group Python 3 compatible
This script only runs on the KVM Hypervisors and these all support
Python 3.
As Python 2 is deprecated at the end of 2019 we need to fix these
scripts to work under Python 3.
CentOS 7, 8 and Ubuntu 16.04 and 18.04 all have Python 3 installed
by default.
Ubuntu 20.04 will no longer have Python 2 installed and therefor
this script needs to be modified to work with Python 3.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* Add dependency of python3 in packaging/centos7/cloud.spec
Using this tool on a hypervisor admins can query KVM Instances running
on that hypervisor if they have the Qemu Guest Agent installed.
All System VMs have this and they can be queried.
For example:
$ cloudstack-guest-tool i-2-25-VM
This will print some information about network and filesystem status.
root@hv-138-a05-23:~# ./cloudstack-guest-tool s-11-VM --command info|jq
{
"network": [
{
"ip-addresses": [
{
"prefix": 8,
"ip-address": "127.0.0.1",
"ip-address-type": "ipv4"
}
],
"name": "lo",
"hardware-address": "00:00:00:00:00:00"
},
{
"ip-addresses": [
{
"prefix": 16,
"ip-address": "169.254.242.169",
"ip-address-type": "ipv4"
}
],
"name": "eth0",
"hardware-address": "0e:00:a9:fe:f2:a9"
},
...
...
"filesystem": [
{
"mountpoint": "/var",
"disk": [
{
"bus": 0,
"bus-type": "virtio",
"target": 0,
"unit": 0,
"pci-controller": {
"slot": 7,
"bus": 0,
"domain": 0,
"function": 0
}
}
],
"type": "ext4",
"name": "vda6"
},
Signed-off-by: Wido den Hollander <wido@widodh.nl>
This may slow down CI and release, but ensures that unit tests always
run as part of the packaging build process.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* systemd: fix services to allow TLS configurations via java.security.ciphers
This fixes the management server and systemd services to allow the
java.security.ciphers file to configure disabled TLS protocols and
algorithms. This also cleans up systemd service files for agent and
usage server.
This fixes#3140
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* configure: fix travis failure due pycodestyle error
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Install CentOS 7 e.g. Build 1804 and Java build 1.8.0_181
if you inspect systemd in debug mode you will see some errors
1.
permission of the cloudstack-managment.service are not corretly set
2.
invalid classpath specified. it seems the string which is used will be divided... we now we use ${..} like the lines above ... confused
Install any additional plugin jars in the lib directory to be picked up
by the classpath builder, otherwise one has to manually add the jar
to /etc/default/cloudstack-management after installation. This fixes
the issue for `mysql-ha` plugin.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This makes the agent logrotate config to `noreplace` so on upgrade
any changes to the file are not lost.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* packaging: use libuuid x86_64 package for cloudstack-common
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* 64 bit links is packaged
* post scan filter to exclude libuuid.so.1
* Revert "packaging: use libuuid x86_64 package for cloudstack-common"
This reverts commit b3fb8957fe4e98c85949be2010f0316c89d535a9.
* post scan filter to exclude libuuid.so.1 (centos63)
* revert removal of 32 bit support for vhd-util libs
This introduces a new global setting `vm.configdrive.primarypool.enabled` to toggle creation/hosting of config drive iso files on primary storage, the default will be false causing them to be hosted on secondary storage. The current support is limited from hypervisor resource side and in current implementation limited to `KVM` only. The next big change is that config drive is created at a temporary location by management server and shipped to either KVM or SSVM agent via cmd-answer pattern, the data of which is not logged in logs. This saves us from adding genisoimage dependency on cloudstack-agent pkg.
The APIs to reset ssh public key, password and user-data (via update VM API) requires that VM should be shutdown. Therefore, in the refactoring I removed the case of updation of existing ISO. If there are objections I'll re-put the strategy to detach+attach new config iso as a way of updation. In the refactored implementation, the folder name is changed to lower-cased configdrive. And during VM start, migration or shutdown/removal if primary storage is enable for use, the KVM agent will handle cleanup tasks otherwise SSVM agent will handle them.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Remove deprecated tomcat configuration file instead of moving it, since moving it prevented the 4.11 upgrade from overwriting it with the new jetty configurations. Also added systemctl daemon-reload to refresh daemon services so that management server can start up seamlessly after the upgrade.
Fixes#2537
