137 Commits

Author SHA1 Message Date
Wei Zhou
3f2449f3d8
packaging: install tzdata-java on centos7/centos8 (#7768)
When install cloudstack on latest rocky8 server with java-11-openjdk-11.0.20.0.8-2.el8.x86_64, management server cannot be started due to error
```
Caused by: java.lang.Error: java.io.FileNotFoundException: /usr/lib/jvm/java-11-openjdk-11.0.20.0.8-2.el8.x86_64/lib/tzdb.dat (No such file or directory)
```

It is because the package `tzdata-java` is not automatically installed.
(It was installed with java-11-openjdk-11.0.18.0.10-2.el8_7.x86_64, but not with latest java-11-openjdk-11.0.20.0.8-2.el8.x86_64)
2023-07-26 03:20:53 +02:00
Wei Zhou
62e342c1bc
utils,framework/db: Introduce new database encryption cipher based on AesGcmJce (#7003) 2023-02-02 16:25:49 +01:00
dahn
cdaad257ea
resolve sanity check last id file acces problems (#6825)
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2022-12-21 17:10:14 +01:00
Marcus Sorensen
289a43f758
packaging: Uses rng-tools for entropy, not haveged for cloudstack-management (#6775)
* EL8 uses rng-tools, not haveged, to gather entropy

* packaging: use rng-tools/rngd consistently across all distros

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-10-14 16:51:17 +05:30
Ann
0400765e1f
packaging: rpm dependencies: which and file (#6638)
createtmplt.sh will use file, and setup-sysvm-tmplt will use which. this is log:

/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt: line 74: which: command not found
/usr/share/cloudstack-common/scripts/storage/secondary/createtmplt.sh: line 56: file: command not found
/usr/share/cloudstack-common/scripts/storage/secondary/createtmplt.sh: line 75: file: command not found
/usr/share/cloudstack-common/scripts/storage/secondary/createtmplt.sh: line 42: file: command not found
/usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt: line 74: which: command not found
/usr/share/cloudstack-common/scripts/storage/secondary/createtmplt.sh: line 56: file: command not found
/usr/share/cloudstack-common/scripts/storage/secondary/createtmplt.sh: line 75: file: command not found
/usr/share/cloudstack-common/scripts/storage/secondary/createtmplt.sh: line 42: file: command not found
when no which and file, systemvm template can not be import correctly(not decompressed, stay bzip2), then libvirtd will report error:

Could not open backing file: Image is not in qcow2 format
Then systemvm can not be started.
2022-10-08 12:08:25 +05:30
Peinthor Rene
ff961c9594
linstor: support QoS(IOPs) and small improvements (#6682)
This PR has 3 improvements for the Linstor primary storage driver:

- Create a separate jar of it and move all Linstor related classes into the correct project (similar to the storpool plugin)
- Add aux properties for Cloudstack volumes in Linstor to make it easier to identify them in Linstor
- Add support for IOPs settings with the Linstor storage plugin
2022-10-08 12:06:49 +05:30
Marcus Sorensen
697e12f8f7
kvm: volume encryption feature (#6522)
This PR introduces a feature designed to allow CloudStack to manage a generic volume encryption setting. The encryption is handled transparently to the guest OS, and is intended to handle VM guest data encryption at rest and possibly over the wire, though the actual encryption implementation is up to the primary storage driver.

In some cases cloud customers may still prefer to maintain their own guest-level volume encryption, if they don't trust the cloud provider. However, for private cloud cases this greatly simplifies the guest OS experience in terms of running volume encryption for guests without the user having to manage keys, deal with key servers and guest booting being dependent on network connectivity to them (i.e. Tang), etc, especially in cases where users are attaching/detaching data disks and moving them between VMs occasionally.

The feature can be thought of as having two parts - the API/control plane (which includes scheduling aspects), and the storage driver implementation.

This initial PR adds the encryption setting to disk offerings and service offerings (for root volume), and implements encryption support for KVM SharedMountPoint, NFS, Local, and ScaleIO storage pools.

NOTE: While not required, operations can be significantly sped up by ensuring that hosts have the `rng-tools` package and service installed and running on the management server and hypervisors. For EL hosts the service is `rngd` and for Debian it is `rng-tools`. In particular, the use of SecureRandom for generating volume passphrases can be slow if there isn't a good source of entropy. This could affect testing and build environments, and otherwise would only affect users who actually use the encryption feature. If you find tests or volume creates blocking on encryption, check this first.

### Management Server

##### API

* createDiskOffering now has an 'encrypt' Boolean
* createServiceOffering now has an 'encryptroot' Boolean. The 'root' suffix is added here in case there is ever any other need to encrypt something related to the guest configuration, like the RAM of a VM.  This has been refactored to deal with the new separation of service offering from disk offering internally.
* listDiskOfferings shows encryption support on each offering, and has an encrypt boolean to choose to list only offerings that do or do not support encryption
* listServiceOfferings shows encryption support on each offering, and has an encrypt boolean to choose to list only offerings that do or do not support encryption
* listHosts now shows encryption support of each hypervisor host via `encryptionsupported`
* Volumes themselves don't show encryption on/off, rather the offering should be referenced. This follows the same pattern as other disk offering based settings such as the IOPS of the volume.

##### Volume functions

A decent effort has been made to ensure that the most common volume functions have either been cleanly supported or blocked. However, for the first release it is advised to mark this feature as *experimental*, as the code base is complex and there are certainly edge cases to be found.

Many of these features could eventually be supported over time, such as creating templates from encrypted volumes, but the effort and size of the change is already overwhelming.

Supported functions:
* Data Volume create
* VM root volume create
* VM root volume reinstall
* Offline volume snapshot/restore
* Migration of VM with storage (e.g. local storage VM migration)
* Resize volume
* Detach/attach volume

Blocked functions:
* Online volume snapshot
* VM snapshot w/memory
* Scheduled snapshots (would fail when VM is running)
* Disk offering migration to offerings that don't have matching encryption
* Creating template from encrypted volume
* Creating volume from encrypted volume
* Volume extraction (would we decrypt it first, or expose the key? Probably the former).

##### Primary Storage Support

For storage developers, adding encryption support involves:

1. Updating the `StoragePoolType` for your primary storage to advertise encryption support. This is used during allocation of storage to match storage types that support encryption to storage that supports it.

2. Implementing encryption feature when your `PrimaryDataStoreDriver` is called to perform volume lifecycle functions on volumes that are requesting encryption. You are free to do what your storage supports - this could be as simple as calling a storage API with the right flag when creating a volume. Or (as is the case with the KVM storage types), as complex as managing volume details directly at the hypervisor host. The data objects passed to the storage driver will contain volume passphrases, if encryption is requested.

##### Scheduling

For the KVM implementations specified above, we are dependent on the KVM hosts having support for volume encryption tools. As such, the hosts `StartupRoutingCommand` has been modified to advertise whether the host supports encryption. This is done via a probe during agent startup to look for functioning `cryptsetup` and support in `qemu-img`. This is also visible via the listHosts API and the host details in the UI.  This was patterned after other features that require hypervisor support such as UEFI.

The `EndPointSelector` interface and `DefaultEndpointSelector` have had new methods added, which allow the caller to ask for endpoints that support encryption.  This can be used by storage drivers to find the proper hosts to send storage commands that involve encryption. Not all volume activities will require a host to support encryption (for example a snapshot backup is a simple file copy), and this is the reason why the interface has been modified to allow for the storage driver to decide, rather than just passing the data objects to the EndpointSelector and letting the implementation decide.

VM scheduling has also been modified. When a VM start is requested, if any volume that requires encryption is attached, it will filter out hosts that don't support encryption.

##### DB Changes

A volume whose disk offering enables encryption will get a passphrase generated for it before its first use. This is stored in the new 'passphrase' table, and is encrypted using the CloudStack installation's standard configured DB encryption. A field has been added to the volumes table, referencing this passphrase, and a foreign key added to ensure passphrases that are referenced can't be removed from the database.  The volumes table now also contains an encryption format field, which is set by the implementer of the encryption and used as it sees fit.

#### KVM Agent

For the KVM storage pool types supported, the encryption has been implemented at Qemu itself, using the built-in LUKS storage support. This means that the storage remains encrypted all the way to the VM process, and decrypted before the block device is visible to the guest.  This may not be necessary in order to implement encryption for /your/ storage pool type, maybe you have a kernel driver that decrypts before the block device on the system, or something like that. However, it seemed like the simplest, common place to terminate the encryption, and provides the lowest surface area for decrypted guest data.

For qcow2 based storage, `qemu-img` is used to set up a qcow2 file with LUKS encryption. For block based (currently just ScaleIO storage), the `cryptsetup` utility is used to format the block device as LUKS for data disks, but `qemu-img` and its LUKS support is used for template copy.

Any volume that requires encryption will contain a passphrase ID as a byte array when handed down to the KVM agent. Care has been taken to ensure this doesn't get logged, and it is cleared after use in attempt to avoid exposing it before garbage collection occurs.  On the agent side, this passphrase is used in two ways:

1. In cases where the volume experiences some libvirt interaction it is loaded into libvirt as an ephemeral, private secret and then referenced by secret UUID in any libvirt XML. This applies to things like VM startup, migration preparation, etc.

2. In cases where `qemu-img` needs to use this passphrase for volume operations, it is written to a `KeyFile` on the cloudstack agent's configured tmpfs and passed along. The `KeyFile` is a `Closeable` and when it is closed, it is deleted. This allows us to try-with-resources any volume operations and get the KeyFile removed regardless.

In order to support the advanced syntax required to handle encryption and passphrases with `qemu-img`, the `QemuImg` utility has been modified to support the new `--object` and `--image-opts` flags. These are modeled as `QemuObject` and `QemuImageOptions`.  These `qemu-img` flags have been designed to supersede some of the existing, older flags being used today (such as choosing file formats and paths), and an effort could be made to switch over to these wholesale. However, for now we have instead opted to keep existing functions and do some wrapping to ensure backward compatibility, so callers of `QemuImg` can choose to use either way.

It should be noted that there are also a few different Enums that represent the encryption format for various purposes. While these are analogous in principle, they represent different things and should not be confused. For example, the supported encryption format strings for the `cryptsetup` utility has `LuksType.LUKS` while `QemuImg` has a `QemuImg.PhysicalDiskFormat.LUKS`.

Some additional effort could potentially be made to support advanced encryption configurations, such as choosing between LUKS1 and LUKS2 or changing cipher details. These may require changes all the way up through the control plane. However, in practice Libvirt and Qemu currently only support LUKS1 today. Additionally, the cipher details aren't required in order to use an encrypted volume, as they're stored in the LUKS header on the volume there is no need to store these elsewhere.  As such, we need only set the one encryption format upon volume creation, which is persisted in the volumes table and then available later as needed.  In the future when LUKS2 is standard and fully supported, we could move to it as the default and old volumes will still reference LUKS1 and have the headers on-disk to ensure they remain usable. We could also possibly support an automatic upgrade of the headers down the road, or a volume migration mechanism.

Every version of cryptsetup and qemu-img tested on variants of EL7 and Ubuntu that support encryption use the XTS-AES 256 cipher, which is the leading industry standard and widely used cipher today (e.g. BitLocker and FileVault).

Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
2022-09-27 10:20:59 +05:30
Pearl Dsilva
830f3061bc
SystemVM optimizations (#5831)
* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* refactor patching logic - certs

* move logic of fixing rootdiskcontroller from upgrade to kubernetes service

* add livepatch option to restart network & vpc

* smooth upgrade of cks clusters

* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* add cgroup config for containerd

* add systemd config for kubelet

* add additional info during image registry config

* address comments

* add temp links of download.cloudstack.org

* address part of the comments

* address comments

* update containerd config - as version has upgraded to 1.5 from 1.4.12 in 4.17.0

* address comments - simplify

* fix vue3 related icon changes

* allow network commands when router template version is lower but is patched

* add internal LB to the list of routers to be patched on network restart with live patch

* add unit tests for API param validations and new helper utilities - file scp & checksum validations

* perform patching only for non-user i.e., system VMs

* add test to validate params

* remove unused import

* add column to domain_router to display software version and support networkrestart with livePatch from router view

* Requires upgrade column to consider package (cloud-scripts) checksum to identify if true/false

* use router software version instead of checksum

* show N/A if no software version reported i.e., in upgraded envs

* fix deb failure

* update pom to official links of systemVM template
2022-04-21 13:40:19 -03:00
slavkap
4004dfcfd8
StorPool storage plugin (#6007)
* StorPool storage plugin

Adds volume storage plugin for StorPool SDS

* Added support for alternative endpoint

Added option to switch to alternative endpoint for SP primary storage

* renamed all classes from Storpool to StorPool

* Address review

* removed unnecessary else

* Removed check about the storage provider

We don't need this check, we'll get if the snapshot is on StorPool be
its name from path

* Check that current plugin supports all functionality before upgrade CS

* Smoke tests for StorPool plug-in

* Fixed conflicts

* Fixed conflicts and added missed Apache license header

* Removed whitespaces in smoke tests

* Added StorPool plugin jar for Debian

the StorPool jar will be included into cloudstack-agent package for
Debian/Ubuntu
2022-04-14 11:12:01 -03:00
Daniel Augusto Veronezi Salvador
c146449ab8
Create profiles to download systemvm-templates (#5602)
* Create profiles to download systemvm-templates

* Rename profiles

* Add support to pass necessary flags to the packaging jobs

* Escape flags

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
2022-03-17 00:55:27 -03:00
Wei Zhou
6495bc1a47
packaging: display First Install and Onboarding Message (#5851)
* packaging: display First Install and Onboarding Message

* Update #5851: Update as per Rohit's comments

* Update #5851: display package name in help message

* Update #5851: display links of installed cloudstack version on UI

* Update #5851: fix vue warnings
2022-02-08 16:11:03 +01:00
Rohit Yadav
28f6067f3c
packaging: use modern systemctl enable/disable (#5881)
This fixes spec files to use systemctl enable|disable instead of on|off.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-01-27 09:53:16 +05:30
davidjumani
6ac834a358
Adding AutoScaling for cks + CKS CoreOS EOL update + systemvmtemplate improvements (#4329)
Adding AutoScaling support for cks
Kubernetes PR : kubernetes/autoscaler#3629
Also replaces CoreOS with Debian
Fixes #4198

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-10-06 21:17:41 +05:30
Greg Goodrich
2bbc78170b
packaging: Issue #3974 Deploying mysql-ha jar file into its own path to allow for easy inclusion of just this jar file into multiple packages, such as cloudstack-management and cloudstack-usage. (#3975)
Support for centos7 and centos8

Prior PR #2915 causes the cloudstack-usage daemon to not be able to find the mysql-ha jar file.

This happens because the /etc/default/cloudstack-usage file points to the old location for the mysql-ha jar file, and thus is unable to load it. This prior PR installs this jar into a more common area with other jar files, and thus is not really a desired location for the cloudstack-usage daemon to get it from, as it will cause cloudstack-usage to load other plugins that it doesn't need.

Fixes: #3974
2021-09-08 10:31:11 +05:30
sureshanaparti
0d72886ee3
Replace the DB properties having master and slave(s), with source and replica(s) respectively, on Upgrade for inclusiveness (#5162)
Updates the DB properties (with strict / full property string search) having master and slave(s), with source and replica(s) respectively on upgrade (for inclusiveness).

Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2021-07-01 10:12:59 +05:30
Rohit Yadav
f58b72f6f7 Merge remote-tracking branch 'origin/4.15' 2021-06-27 18:25:46 +05:30
davidjumani
5195f8b4b7
packaging: Create cloud user and group if not present (#5115) 2021-06-24 11:08:39 +05:30
dahn
a1f825e7c7
python3: Migrate Marvin and smoketests to python3 (#4727)
This PR prepares marvin and tests for python3. it was part of #4479, until nose2 was decided to be abandoned from that PR.

Re-PR of #4543 and #3730 to enable cooperation

Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Gabriel Beims Bräscher <gabriel@apache.org>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-05-04 23:19:37 +05:30
Rohit Yadav
abec6d1fff
packaging: build and bundle UI using npm in deb and rpm packages (#4605)
This changes deb and rpm packaging to build the UI using npm and bundle
it in the `cloudstack-management` package and a new `cloudstack-ui`
package. The `cloudstack-ui` package will install the UI under
`/usr/share/cloudstack-ui/`. For both packages the config.json will not
be overridden on upgrade and hosted at /etc/cloudstack/management
for the cloudstack-mangement package, and at /etc/cloudstack/ui for the
cloudstack-ui package. The cloudstack-ui package is for advanced users
who only want the UI want to setup reverse proxy (separate hosting of UI).

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-01-28 14:29:31 +05:30
Andrija Panic
238eccc317
packaging: Minor message update (#4333)
adding quotes, to fix the "servers" to "server's"
2020-09-21 14:12:52 +05:30
Andrija Panic
cddf69c0f8
Usage-server update message improvement (#4278)
* Update cloud.spec

* Update cloud.spec

* Update cloudstack-usage.postinst
2020-08-23 16:04:38 +05:30
davidjumani
d949302d0f
packaging: Adding Centos8, Ubuntu 20.04, XCPNG8.1 Support (#4068)
* DB : Add support for MySQL 8

- Splits commands to create user and grant access on database, the old
statement is no longer supported by MySQL 8.x
- `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove
that from db.properties conn parameters

For mysql-server 8.x setup the following changes were added/tested to
make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and
then restart the mysql-server process:

    server_id = 1
    sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
    innodb_rollback_on_timeout=1
    innodb_lock_wait_timeout=600
    max_connections=1000
    log-bin=mysql-bin
    binlog-format = 'ROW'

    default-authentication-plugin=mysql_native_password

Notice the last line above, this is to reset the old password based
authentication used by MySQL 5.x.

Developers can set empty password as follows:

    > sudo mysql -u root
    ALTER USER 'root'@'localhost' IDENTIFIED BY '';

In libvirt repository, there are two related commits

2019-08-23 13:13 Daniel P. Berrangé            ● rpm: don't enable socket activation in upgrade if --listen present
2019-08-22 14:52 Daniel P. Berrangé            ● remote: forbid the --listen arg when systemd socket activation

In libvirt.spec.in

        /bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :

Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-08-17 16:28:30 +05:30
harikrishna-patnala
a279d5c453
logging: Logging framework to use only log4j (#4003)
Currently CloudStack is using logging frameworks as log4j and Java util logging, logging wrappers as slf4j and Apache common logging.
Here changes are to made it uniform, using only log4j framework.
Removed Java util logging, slf4j and Apache common logging.
2020-06-17 07:11:23 +05:30
NuxRo
db55910f6b
packaging: missing python3 libvirt dependency for CentOS7 (#4124)
Missing python3 libvirt bindings on CentOS7 effectively break security groups.
There are 0 firewall rules added. The agent logs report:

```2020-06-02 10:58:34,346 DEBUG [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Traceback (most recent call last):  File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 26, in <module>    import libvirtModuleNotFoundError: No module named 'libvirt'
```

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-06-04 07:09:14 +05:30
Nicolas Vazquez
efe00aa7e0
[KVM] Rolling maintenance (#3610) 2020-03-12 16:59:46 +01:00
Rohit Yadav
d90341ebf1
cloudstack: add JDK11 support (#3601)
This adds support for JDK11 in CloudStack 4.14+:

- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-02-12 12:58:25 +05:30
Rohit Yadav
493415d91b
packaging: install python-dnspython or python-dns to fix issue with cloudstack-setup-management (#3854) (#3869)
Fixes #3817 #3841

Co-authored-by: Wei Zhou <ustcweizhou@gmail.com>
2020-02-07 11:13:57 +05:30
Anurag Awasthi
1771727ce1 Revert "Extract systemvm.iso using bsdtar (#3536)" (#3800)
This reverts commit 8a55c931e806a64f7f00153d4a58c4463caa7a47.
2020-01-13 12:24:43 +01:00
Kai Takahashi
8a55c931e8 Extract systemvm.iso using bsdtar (#3536)
* Extract systemvm.iso using bsdtar if available.

Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>

* New dependency for CentOS 7 and Debian: bsdtar

bsdtar can extract iso images without mounting.

Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>

* Remove all 'mount' and 'umount' command call(s).

Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
2020-01-06 11:46:20 +01:00
Andrija Panic
45503e104a
Revert "Revert "Fix virtual template size for managed storage for KVM / refactor cloud-install-sys-tmplt (#3371)" (#3771)" (#3772)
This reverts commit e319c8b8f0e742b4f6139603d7881974d0a02e8f.
2019-12-19 19:50:30 +01:00
Andrija Panic
e319c8b8f0
Revert "Fix virtual template size for managed storage for KVM / refactor cloud-install-sys-tmplt (#3371)" (#3771)
This reverts commit a0efbf9d09e57a907cd599d28a1def02686240f1.
2019-12-17 15:29:37 +01:00
Sven Vogel
a0efbf9d09
Fix virtual template size for managed storage for KVM / refactor cloud-install-sys-tmplt (#3371)
* remove unused chksum flag
* refactor cloud-install-sys-tmplt / add qemu-img command to calculate correct virtual size
* add qemu-utils dependency to debian build control file
* add qemu-utils dependency to centos spec file
2019-12-17 15:27:03 +01:00
Rohit Yadav
e93f6bf3c2
utils: use iproute to get default network interface (#3704)
Use iproute to get default network interface

This uses `ip route` than `route` without an explicit path (/bin, /sbin
etc.) to query the default network interface on the system.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-11-27 09:35:22 +05:30
fc9a525adc New BuildRequires for CentOS 7: python-setuptools (#3630)
(It's already added for debian from
commit 4a77a799490a0cd655b1a53377c4db1bb51ddc7c)

Signed-off-by: Kai Takahashi <www.carrotsoft@gmail.com>
2019-11-21 11:32:20 +05:30
Wido den Hollander
899eab66c1 kvm/security_group: Make Security Group Python 3 compatible (#3589)
* kvm/security_group: Make Security Group Python 3 compatible

This script only runs on the KVM Hypervisors and these all support
Python 3.

As Python 2 is deprecated at the end of 2019 we need to fix these
scripts to work under Python 3.

CentOS 7, 8 and Ubuntu 16.04 and 18.04 all have Python 3 installed
by default.

Ubuntu 20.04 will no longer have Python 2 installed and therefor
this script needs to be modified to work with Python 3.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* Add dependency of python3 in packaging/centos7/cloud.spec
2019-09-26 18:17:03 +05:30
Wido den Hollander
8da0556f63 kvm/cloudstack-guest-tool: Tool to query Qemu Guest Agent (#3519)
Using this tool on a hypervisor admins can query KVM Instances running
on that hypervisor if they have the Qemu Guest Agent installed.

All System VMs have this and they can be queried.

For example:

$ cloudstack-guest-tool i-2-25-VM

This will print some information about network and filesystem status.

root@hv-138-a05-23:~# ./cloudstack-guest-tool s-11-VM --command info|jq
{
  "network": [
    {
      "ip-addresses": [
        {
          "prefix": 8,
          "ip-address": "127.0.0.1",
          "ip-address-type": "ipv4"
        }
      ],
      "name": "lo",
      "hardware-address": "00:00:00:00:00:00"
    },
    {
      "ip-addresses": [
        {
          "prefix": 16,
          "ip-address": "169.254.242.169",
          "ip-address-type": "ipv4"
        }
      ],
      "name": "eth0",
      "hardware-address": "0e:00:a9:fe:f2:a9"
    },
 ...
 ...
  "filesystem": [
    {
      "mountpoint": "/var",
      "disk": [
        {
          "bus": 0,
          "bus-type": "virtio",
          "target": 0,
          "unit": 0,
          "pci-controller": {
            "slot": 7,
            "bus": 0,
            "domain": 0,
            "function": 0
          }
        }
      ],
      "type": "ext4",
      "name": "vda6"
    },

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-09-10 09:31:03 -07:00
Sven Vogel
9d02b3679b packaging: set correct logrotate file permissions for cloudstack-agent (#3347)
Correct the permissions on cloudstack-agent logrotate file (/etc/logrotate.d/).
2019-05-24 12:43:08 +05:30
Rohit Yadav
9488c6dcc5 Merge remote-tracking branch 'origin/4.11' into 4.12 2019-04-09 14:22:18 +05:30
Rohit Yadav
96611fc640
packaging: systemctl daemon-reload after agent install or upgrade (#3269)
This runs systemctl daemon-reload after cloudstack-agent is installed
or upgraded.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-04-09 14:21:09 +05:30
Rohit Yadav
55efaf14d9
packaging: don't skip unit tests while building packages (#3266)
This may slow down CI and release, but ensures that unit tests always
run as part of the packaging build process.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-04-08 13:51:30 +05:30
GabrielBrascher
3f17671449 Fix conflict and merge forward PR #3163 from 4.11 to master (4.12)
# Conflicts:
#	packaging/debian/init/cloud-management
#	packaging/systemd/cloudstack-agent.default
#	packaging/systemd/cloudstack-agent.service
#	packaging/systemd/cloudstack-management.service
2019-02-04 23:53:19 -02:00
Rohit Yadav
cb3fed0e4e systemd: fix services to allow TLS configurations via java.security.ciphers (#3163)
* systemd: fix services to allow TLS configurations via java.security.ciphers

This fixes the management server and systemd services to allow the
java.security.ciphers file to configure disabled TLS protocols and
algorithms. This also cleans up systemd service files for agent and
usage server.

This fixes #3140

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* configure: fix travis failure due pycodestyle error

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-02-04 19:51:30 -02:00
Rohit Yadav
52f68a273a Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-12-04 16:39:21 +05:30
Sven Vogel
17097929b6 packaging: correct permissions in spec file and fix class path specified variable (#3030)
Install CentOS 7 e.g. Build 1804 and Java build 1.8.0_181

if you inspect systemd in debug mode you will see some errors
1.
permission of the cloudstack-managment.service are not corretly set
2.
invalid classpath specified. it seems the string which is used will be divided... we now we use ${..} like the lines above ... confused
2018-12-01 01:38:01 +05:30
Rohit Yadav
b8ed159f47 Merge remote-tracking branch 'origin/4.11' 2018-10-25 08:14:49 +05:30
Rohit Yadav
9b35b64b3c
packaging: install plugins at /usr/share/cloudstack-management/lib (#2915)
Install any additional plugin jars in the lib directory to be picked up
by the classpath builder, otherwise one has to manually add the jar
to /etc/default/cloudstack-management after installation. This fixes
the issue for `mysql-ha` plugin.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-24 18:21:03 +05:30
Rohit Yadav
b6302d4e90 Merge remote-tracking branch 'origin/4.11'
Conflicts resolved for:
	engine/orchestration/src/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java
	engine/schema/src/com/cloud/vm/dao/UserVmDaoImpl.java
server/src/com/cloud/network/element/VirtualRouterElement.java
server/src/com/cloud/vm/UserVmManagerImpl.java
tools/marvin/setup.py

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-05 05:18:42 +05:30
Rohit Yadav
c2f4b3653d
packaging: Fixes #2857 don't overwrite agent logrotate config (#2860)
This makes the agent logrotate config to `noreplace` so on upgrade
any changes to the file are not lost.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-09-25 11:25:06 -04:00
Rohit Yadav
85750f918b Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-06-20 12:31:52 +05:30
Rohit Yadav
c87ca1b262 packaging: use libuuid x86_64 package for cloudstack-common (#2706)
* packaging: use libuuid x86_64 package for cloudstack-common

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* 64 bit links is packaged

* post scan filter to exclude libuuid.so.1

* Revert "packaging: use libuuid x86_64 package for cloudstack-common"

This reverts commit b3fb8957fe4e98c85949be2010f0316c89d535a9.

* post scan filter to exclude libuuid.so.1 (centos63)

* revert removal of 32 bit support for vhd-util libs
2018-06-19 13:34:44 +02:00