456 Commits

Author SHA1 Message Date
Wei Zhou
90526419a3 fix build error after merge-forward 2024-02-13 10:40:15 +01:00
Wei Zhou
93406f5582 Merge remote-tracking branch 'apache/4.19' 2024-02-13 10:38:21 +01:00
Rene Peinthor
70b634fff2
Linstor: add HA support and small cleanups (#8407)
* linstor: Outline get storagepools from resourcegroup into function

* linstor: move getHostname() to kvm/Pool and reimplement

* linstor: implement CloudStack HA support
2024-02-13 11:16:12 +05:30
Vishesh
9b18243b87
Fix merge forward bug (#8636) 2024-02-09 12:19:03 +01:00
Wei Zhou
af2e277999 Merge remote-tracking branch 'apache/4.18' into 4.19 2024-02-09 11:53:39 +01:00
Rene Peinthor
393f3d7727
linstor: use relative hostname path (#8633)
As described in issue #8310 some older distributions don't have
hostname in /usr/bin so rely on PATH resolving
2024-02-09 11:49:20 +01:00
Rohit Yadav
cfb4d4378f Merge remote-tracking branch 'origin/4.19'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-02-09 00:13:23 +05:30
Rohit Yadav
a1f547a011 Merge remote-tracking branch 'origin/4.18' into 4.19
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Conflicts:
	plugins/storage/volume/linstor/src/main/java/org/apache/cloudstack/storage/datastore/util/LinstorUtil.java
2024-02-09 00:10:34 +05:30
slavkap
1d1b332141
remove StorPool tags from detached volumes (#8377)
* remove tags from detached volumes

* Adress comments

* address comments

* Address comments
2024-02-09 00:05:34 +05:30
Rene Peinthor
56f0448f0d
Linstor fix migration while node offline (#8610)
* linstor: Add util method getBestErrorMessage from main

* linstor: failed remove of allow-two-primaries is no fatal error

* linstor: Fix failure if a Linstor node is down while migrating

If a Linstor node is down while migrating resource, allow-two-primaries
setting will fail because we can't reach the downed node. But it will
still set the property on the other nodes and migration should work.
We now just report an error instead of completely failing.
2024-02-08 23:57:38 +05:30
João Jandre
49cecaed06
Normalize loggers and upgrade log4j 1.2 to log4j 2.19 (#7131)
* Normalize logs

All classes that could have their loggers inherited from their fathers had their own loggers deleted;
Most loggers didn't have to be static, so most of them were normalized so that they wouldn't be;
All loggers are protected now;
Static logger's name are now 'LOGGER';
Non-static logger's name are now 'logger';
New class DbUpgradeAbstractImpl created so that all Upgraders extend it and inherit its logger

* Upgrade log4j

* fix errors caused by the merge

* Refactor cglibThrowableRenderer functionality to log4j2 and upgrade the last configuration files

* fix sonarcloud bug

* Fix errors caused by merge, remove some unused loggers, and rename a variable that was mistakenly renamed on the normalization commit

* Readd snmpTrapAppender, remove TestAppender

* Regenerate changes

* regenerate changes

* refactor last custom appender

* fix systemvm configuration xml

* Regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* Fix utils pom

* fix some tests

* regenerate changes

* Fix jar being printed on exception

* fix logging in system VMs, fix commands not having log4j2 classpath.

* regenerate changes

* Fix some unwanted renomeations

* fix end of file

* regenerate changes

* regenerate changes

* fix merge error

* regenerate changes

* fix tests

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* readd reload4j to tungsten as juniper depends on it

* Regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* re-add reload4j dependency to network-contrail, as juniper depends on it

* regenerate changes

* regenerate changes

* regenerate changes

* fix typo

* regenerate changes

* regenerate changes

* Fix end of files

* regenerate changes

* add logj42 to cloud-utils-SHADED.jar

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* Regenerate changes

* Regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* Regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* Regenerate changes

* fix some tests

* Regenerate changes

* Regenerate changes

* fix test

* Regenerate changes

* Regenerate changes
2024-02-08 09:55:41 -03:00
Vishesh
399bd0a067
Upgrade to mockito 4 and handle Mockito deprecations (#8427) 2024-02-06 14:20:37 +01:00
Rohit Yadav
25009bfb76 Merge remote-tracking branch 'origin/4.19' 2024-02-05 14:22:09 +05:30
slavkap
94c8b1da5c
Option to create StorPool primary storage with a valid URL (#8356)
* Option to create primary storage with a valid URL

* check if the scheme is valid
2024-02-05 14:21:13 +05:30
Suresh Kumar Anaparti
8ea9fc911d
StoragePoolType as class (#8544)
* StoragePoolType as a class

* Fix agent side StoragePoolType enum to class

* Handle StoragePoolType for StoragePoolJoinVO

* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.

* Fix UserVMJoinVO for StoragePoolType

* fixed missing imports

* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.

* Fixed equals for the enum.

* removed not needed try/catch for prepareAttribute

* Added license to the file.

* Implemented "supportsPhysicalDiskCopy" for storage adaptor.

Co-authored-by: mprokopchuk <mprokopchuk@apple.com>

* Add javadoc to StoragePoolType class

* Add unit test for StoragePoolType comparisons

* StoragePoolType "==" and ".equals()" fix.

* Fix StoragePoolType for FiberChannelAdapter

* Fix for abstract storage adaptor set up issue

* review comments

* Pass StoragePoolType object for poolType dao attribute

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@gmail.com>
2024-02-05 13:27:15 +05:30
Abhishek Kumar
7dffbc6e47 Updating pom.xml version numbers for release 4.20.0.0-SNAPSHOT
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-02-02 18:16:37 +05:30
Abhishek Kumar
a7b97ff3b0 Updating pom.xml version numbers for release 4.19.1.0-SNAPSHOT
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-02-02 18:06:04 +05:30
Abhishek Kumar
2746225b99 Updating pom.xml version numbers for release 4.19.0.0
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-01-29 10:21:52 +05:30
Abhishek Kumar
d6ac91f2df
minio: fix store user creation (#8425)
To prevent errors during multi-user access, use account UUID to create/access user on the provider side. Also, update the existing secret key for a user that already exists.
2024-01-09 17:44:11 +05:30
Rene Glover
1031c31e6a
FiberChannel Multipath for KVM + Pure Flash Array and HPE-Primera Support (#7889)
This PR provides a new primary storage volume type called "FiberChannel" that allows access to volumes connected to hosts over fiber channel connections. It requires Multipath to provide path discovery and failover. Second, the PR adds an AdaptivePrimaryDatastoreProvider that abstracts how volumes are managed/orchestrated from the connector to communicate with the primary storage provider, using a ProviderAdapter interface, allowing the code interacting with the primary storage provider API's to be simpler and have no direct dependencies on Cloudstack code. Lastly, the PR provides an implementation of the ProviderAdapter classes for the HP Enterprise Primera line of storage solutions and the Pure Flash Array line of storage solutions.
2023-12-09 11:31:33 +05:30
Abhishek Kumar
c599011ef5 Merge remote-tracking branch 'apache/4.18' 2023-12-08 18:06:15 +05:30
Peinthor Rene
bba554bcc4
linstor: Fix possible NPE if Linstor storage-pool data missing (#8319)
If Linstor doesn't return storage pool info, certain values are null.
Now we assume the values are 0 if we get null values.
2023-12-08 17:02:18 +05:30
Peinthor Rene
a15b706fbe
Linstor: Allow snapshot backup also to work on non hyperconverged setups (#8271)
On no access to the storage nodes, we now create a temporary resource from the snapshot and copy that data into the secondary storage. Revert works the same, just that we now also look additionally for any Linstor agent node.

Also enables now backup snapshot by default.

This whole BackupSnapshot functionality was introduced in 4.19,
so I would be happy if this still could be merged.
2023-12-05 12:59:52 +05:30
kishankavala
5651eab49c
ObjectStore Framework with MinIO and Simulator plugins (#7752)
This PR adds Object Storage feature to CloudStack.

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+CloudStack+Object+Store
2023-12-01 17:51:00 +05:30
João Jandre
26b01f6f3b
Flexible tags for hosts and storage pools (#7489)
Co-authored-by: João Jandre <joao@scclouds.com.br>
2023-11-30 09:36:47 +01:00
DK101010
6001772335
multi local storage handling for kvm (#6699)
Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2023-11-16 16:43:42 +01:00
Peinthor Rene
68e504aff9
Linstor backup snaphots (#8067)
This PR adds an config option for the Linstor primary storage driver, that allows you to automatically backup
volume snapshots to the secondary storage.
Additionally it will not mangle the need java-linstor dependency into the client.jar, but instead just copy
the java-linstor.jar into lib.

Config option is called: lin.backup.snapshots and is default false

The scope of this change should be limited, as it only touches the Linstor driver and a part of copyAsync
was implemented with 2 new Linstor specific commands.
2023-11-09 09:38:10 +05:30
slavkap
2bb182c3e1
KVM Host HA enhancement for StorPool storage (#8045)
Extending the current functionality of KVM Host HA for the StorPool storage plugin and the option for easy integration for the rest of the storage plugins to support Host HA

This extension works like the current NFS storage implementation. It allows it to be used simultaneously with NFS and StorPool storage or only with StorPool primary storage.

If it is used with different primary storages like NFS and StorPool, and one of the health checks fails for storage, there is an option to report the failure to the management with the global config kvm.ha.fence.on.storage.heartbeat.failure. By default this option is disabled when enabled the Host HA service will continue with the checks on the host and eventually will fence the host
2023-11-04 12:35:37 +05:30
John Bampton
f090c77f41
misc: fix spelling (#7549)
Co-authored-by: Stephan Krug <stekrug@icloud.com>
2023-11-02 09:23:53 +01:00
Vishesh
5362bad442
Storage Management (#7949) 2023-11-01 10:46:22 +01:00
Abhishek Kumar
543c54c718
api,server,ui: snapshot copy, multi-zone replica (#7873)
This PR adds new functionality to copy snapshots across zones and take snapshots for multiple zones.

Copy functionality is similar to template copy. The source zone acts as the web server from where the destination zone(s) can download the snapshot files. For this purpose, a new API - `copySnapshot` has been added. The response for copySnapshot will be returning zone and download details from the first destination zone of the request. This behaviour is similar to the `copyTemplate` API.

In a similar manner, multiple zones can be selected while taking the snapshots or creating snapshot policies. For this snapshot will be taken in the base zone(in which volume is present) and then copied to the additional zones. A new parameter - `zoneids` has been added to `createSnapshot` and `createSnapshotPolicy` APIs.

As snapshots can be present on multiple zones (secondary stores), a new parameter `zoneid` has been added to delete the snapshot copy on a specific zone.

`listSnapshots` API has been updated to allow listing snapshot entries for different zones/datastores. New parameters - `showUnique`, `locationType` have been added.

Events generated during snapshot operations will now be linked to the snapshot itself rather than the volume of the snapshot.

`listSnapshotPolicies` and `createSnapshotPolicy` APIs will return zone details of the zones in which backup will be scheduled for the policy.

----
New API added
`copySnapshot`

Request and response params updated for APIs
```
- listSnapshots
- deleteSnapshot
- createTemplate
- listZones
- listSnapshotPolicies
- createSnapshotPolicy
```
UI updated for
- Snapshot detail view
- Create snapshot form
- Create snapshot policy form
- Create volume (from snapshot) form
- Create template (from snapshot) form

Doc PR: https://github.com/apache/cloudstack-documentation/pull/344
PR: https://github.com/apache/cloudstack/pull/7873
2023-10-23 09:01:58 +02:00
Abhishek Kumar
fcbf540369 Merge remote-tracking branch 'apache/4.18' into main 2023-10-19 10:48:00 +05:30
Peinthor Rene
67cb9b9e40
linstor: fix template copy on non hyperconverged setups (#8114)
Making a diskful resource was meant as an optimization,
but cannot work on non hyperconverged setups,
as the storage nodes (diskful) are not part of the cloudstack cluster.
2023-10-19 10:46:20 +05:30
Abhishek Kumar
be039a1e46 Merge remote-tracking branch 'apache/4.18' into main 2023-10-17 17:22:41 +05:30
Peinthor Rene
4a86a0d233
linstor: Fix template volume missing on copy node (#8082)
A TODO was overseen and never implemented,
which could trigger the following bug:

If Linstor didn't create a resource (diskless or diskfull) on
the cloudstack choosen node, it would not be able to copy the
template data there, it even seems no error was
triggered and the new template file silently just became
empty/corrupt.
2023-10-17 17:05:42 +05:30
Daan Hoogland
d627539688 Merge branch '4.18' 2023-10-02 16:03:28 +02:00
Peinthor Rene
96205a51ef
linstor: resize root disk on offerings with different size (#7952) 2023-10-02 15:58:00 +02:00
slavkap
3c608290db
Fix tests on CS build when cloudstack-management service is running (#8009) 2023-09-27 19:32:07 +05:30
Vishesh
84e770bf95
Remove powermock from pom.xml (#8000) 2023-09-26 13:31:56 +02:00
Wei Zhou
246bb24b0f Updating pom.xml version numbers for release 4.18.2.0-SNAPSHOT
Signed-off-by: Wei Zhou <weizhou@apache.org>
2023-09-12 17:26:53 +02:00
Wei Zhou
f6b2a58727 Merge branch '4.18' 2023-09-07 08:56:35 +02:00
Wei Zhou
4bdff06acd Updating pom.xml version numbers for release 4.18.1.0
Signed-off-by: Wei Zhou <weizhou@apache.org>
2023-09-07 08:50:50 +02:00
Marcus Sorensen
89e0a4ca25
PowerFlex: Handle missing volumes gracefully during delete volume (#7924) 2023-08-31 21:43:45 +02:00
Daan Hoogland
24ae5aa5fa Merge branch '4.18' 2023-08-25 14:27:34 +02:00
Rohit Yadav
e964395bd4
vmware: improve solidfire storage plugin integration and fix cases (#3) (#7761)
This fixes the following cases in which Solidfire storage integration
caused issues when using Solidfire datadisks with VMware:

1. Take Volume Snapshot of Solidfire data disk
2. Delete an active Instance with Solidfire data disk attached
3. Attach used existing Solidfire data disk to a running/stopped VM
4. Stop and Start an instance with Solidfire data disks attached
5. Expand disk by resizing Solidfire data disk by providing size
6. Expand disk by changing disk offering for the Solidfire data disk

Additional changes:
- Use VMFS6 as managed datastore type if the host supports
- Refactor detection and splitting of managed storage ds name in storage
  processor
- Restrict storage rescanning for managed datastore when resizing

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-25 11:36:03 +05:30
Daan Hoogland
ea832bce13 Merge branch '4.18' 2023-08-22 11:44:45 +02:00
Sina Kashipazha
d296f54c7f
Api: update command domainId/account descriptions (#7876) 2023-08-20 15:44:31 +02:00
Wei Zhou
540dcc1040 Merge remote-tracking branch 'apache/4.18' 2023-08-11 08:30:07 +02:00
Wei Zhou
d9319e645c
storpool: fix pom.xml which caused docker image build failures (#7845)
steps to reproduce the issue:
- git clone https://github.com/apache/cloudstack.git
- cd cloudstack
- rm -rf .git/
- run `mvn -P developer,systemvm clean install`

Without this PR, it fails with error

```
 > [ 8/10] RUN mvn -Pdeveloper -Dsimulator -DskipTests clean install:
668.1 [ERROR] Failed to execute goal pl.project13.maven:git-commit-id-plugin:4.9.10:revision (get-the-git-infos) on project cloud-plugin-storage-volume-storpool: .git directory is not found! Please specify a valid [dotGitDirectory] in your pom.xml -> [Help 1]
```
2023-08-10 16:28:49 +02:00
Daan Hoogland
7ee697fcc5 Merge release branch 4.18 to main
* 4.18:
  UI: allow new keys for VM details (#7793)
  Refactoring StorPool's smoke tests (#7392)
  UI: decode userdata in EditVM dialog (#7796)
  packaging: unalias cp before package upgrade (#7722)
  make NoopDbUpgrade do a systemvm template check (#7564)
  UI unit test: fix expected values (#7792)
2023-08-03 15:59:01 +02:00