71 Commits

Author SHA1 Message Date
Abhishek Kumar
4400e02a1b
framework/config,server: configkey caching (#472)
Added caching for ConfigKey value retrievals based on the Caffeine
in-memory caching library.
https://github.com/ben-manes/caffeine
Currently, expire time for a cache is 1 minute and each update of the
config key invalidates the cache.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-09-03 15:53:08 +05:30
Vishesh
8be18e587f
FR75 Enforce strict host tag checking (#421)
* Enforce strict host tag checking

* Add e2e tests

* Add more information to error log

* Fix e2e test

* Update global settings descrption

* fixup

* Fix e2e test teardown
2024-06-25 14:38:59 +05:30
Abhishek Kumar
8f88103a29
FR72 - api,server: purge expunged resources (#405)
This PR introduces the functionality of purging removed DB entries for CloudStack entities (currently only for VirtualMachine).
There would be three mechanisms for purging removed resources:
- Background task - CloudStack will run a background task which runs at a defined interval. Other parameters for this task can be controlled with new global settings.
- API - New API `purgeExpungedResources`. It will allow passing the following parameters - resourcetype, batchsize, startdate, enddate
- Config for service offering. Service offerings can be created with purgeresources parameter which would allow purging resources immediately on expunge.

Following new global settings have been added:
- `expunged.resources.purge.enabled`: Default: false. Whether to run a background task to purge the DB records of the expunged resources.
- `expunged.resources.purge.resources`: Default: (empty). A comma-separated list of resource types that will be considered by the background task to purge the DB records of the expunged resources. Currently only VirtualMachine is supported. An empty value will result in considering all resource types for purging.
- `expunged.resources.purge.interval`: Default: 86400. Interval (in seconds) for the background task to purge the DB records of the expunged resources.
- `expunged.resources.purge.delay`: Default: 300. Initial delay (in seconds) to start the background task to purge the DB records of the expunged resources task.
- `expunged.resources.purge.batch.size`: Default: 50. Batch size to be used during purging of the DB records of the expunged resources.
- `expunged.resources.purge.start.time`: Default: (empty). Start time to be used by the background task to purge the DB records of the expunged resources. Use format `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`.
- `expunged.resources.purge.keep.past.days`: Default: 30. The number of days in the past from the execution time of the background task to purge the DB records of the expunged resources for which the expunged resources must not be purged. To enable purging DB records of the expunged resource till the execution of the background task, set the value to zero.
- `expunged.resource.purge.job.delay`: Default: 180. Delay (in seconds) to execute the purging of the DB records of an expunged resource initiated by the configuration in the offering. Minimum value should be 180 seconds and if a lower value is set then the minimum value will be used.

Upstream PRs:
https://github.com/apache/cloudstack/pull/8999
https://github.com/apache/cloudstack-documentation/pull/397

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2024-06-19 12:59:50 +05:30
Vishesh
7fae1fc747
Fix restore VM with allocated root disk (#441)
* Fix restore VM with allocated root disk

* Add e2e test for restore vm

* Add more checks for e2e test
2024-04-29 12:18:55 +05:30
Abhishek Kumar
6a9cdedda4
api,server,ui: tagged resource limits (#327)
Introduces the concept of tagged resource limits. Limits can be enforced on accounts and domains for the deployment of entities for a tagged resource. Current tagged resource limits can be used for the following resource types,

Host limits

    user_vm
    cpu
    memory

Storage limits

    volume
    primary_storage

Following global settings can used to specify tags for which limit needs to be enforced,

    Host: resource.limit.host.tags
    Storage: resource.limit.storage.tags

Option for specifying tagged resource limits and viewing tagged resource usage are made available in the UI.

Enhances use of templatetag for VM deployment and template creation

Adds option to list disk offering with suitability flag for a virtualmachine. A new parameter named virtualmachineid has been added to the listDiskOfferings API which when passed returns suitableforvirtualmachine param in the reponse.
2024-02-07 17:35:15 +05:30
Vishesh
47e53eceed
E2e test resource listing (#363)
* Use dualzones for ci github actions

* Update advdualzone.cfg to be similar to advanced.cfg & fixup test_metrics_api.py

* Add e2e tests for listing of accounts, disk_offerings, domains, hosts, service_offerings, storage_pools, volumes

* Add test for listing volumes with tags filter

* Add check for existing volumes in test_list_volumes

* Wait for volumes to be deleted on cleanup

* Filter out volumes in Destroy state before checking the count of volumes
2024-01-25 14:52:06 +05:30
Wei Zhou
f570934482 .github: run Sonar Check only on PRs from apache/cloudstack branches (#8058)
This PR fixes #8050

(cherry picked from commit 864a195868036b367ccf22ec1f26b08ce7d931d7)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-11 21:02:57 +05:30
Vishesh
811b6293af
.github: Use parallel builds for github actions (#7592) 2023-06-07 13:17:29 +02:00
Abhishek Kumar
44024a8a8b
github-actions: fix ui build workflow (#7522)
package-lock.json is not removed.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-05-12 12:54:23 +05:30
Rohit Yadav
0e4a5a1fa4 .github: restrict sonar and CI actions to only apache/cloudstack
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-05-10 14:07:29 +05:30
Daan Hoogland
66cbe0a051 replace pull_request_target with pull_request 2023-05-09 18:40:20 +02:00
Daan Hoogland
da58a20cc2 repair vi typo 2023-02-27 09:37:32 +01:00
Daan Hoogland
2fc443e909 bash evaluation within quotes 2023-02-26 21:51:56 +01:00
Daan Hoogland
3ee32244d8 bash evaluation as opposed to gha. 2023-02-26 19:32:46 +01:00
Wei Zhou
0457cc559e .github: allow only read permission in sonar-check.yml 2023-02-23 13:47:48 +01:00
Wei Zhou
bc5dbe95e9
.github: use tag name as docker image name for new tags (#7233) 2023-02-20 12:42:57 +01:00
Wei Zhou
ea18b86684
.github: disable Sonar check on forks as it requires crendential (#7199) 2023-02-13 11:58:05 +01:00
John Bampton
dcf01f7bef
.github: Skip Coverage Check when forks are updated (#7177) 2023-02-13 09:48:31 +01:00
Harikrishna
a3feccf70c
User two factor authentication (#6924)
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-02-13 09:14:17 +01:00
Wei Zhou
e32016d4a1
.github: set docker repository to apache (#7176)
Now the images are pushed to
https://hub.docker.com/r/asfjenkins/cloudstack-simulator/tags
2023-02-07 13:46:22 +01:00
Wei Zhou
b93608eada
Github Action: Build docker image for cloudstack-simulator (#7122) 2023-02-07 10:37:44 +01:00
dahn
6f6cec5f56
.github: run all gha on ubuntu latest (#7091)
* run all gha on ubuntu latest

* replace latest with 22.04
2023-01-18 10:35:56 +05:30
John Bampton
c0a32d00b7
Use Super-Linter Slim Image (#6610)
Co-authored-by: dahn <daan@onecht.net>
2023-01-12 13:41:28 +01:00
John Bampton
00426ede46
Add pre-commit workflow with 3 Git hooks (#6273)
Co-authored-by: dahn <daan@onecht.net>
2023-01-11 12:25:31 +01:00
Rohit Yadav
c2b75f4c81 README: revert project logo and update README
Happy New Year - this reverts project logo and updates README and the
names of Github Actions badges used on README.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-01-05 17:59:08 +05:30
Rohit Yadav
9a18ea28dd .github: use ubuntu:22.04 for Sonar jacoco coverage checks
Apologies to commit this to main directly, this is because the
pull_request_target requires that Github Actions yaml changes are on the
source repo main branch.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-12-30 15:00:17 +05:30
Wei Zhou
dc40ea130f
UI: fix ui test errors (#7030) 2022-12-29 09:08:54 +01:00
John Bampton
bf5cacb3bb
gha(linters): enable 7 more Python flake8 checks (#7005) 2022-12-21 16:53:35 +01:00
Rohit Yadav
2dc9f1e32c
.github: Migrate Travis to Github Actions and Simulator based CI (#6956)
Runs on default spec:
2-core CPU (x86_64)
7 GB of RAM
14 GB of SSD space

https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-12-21 16:36:52 +05:30
John Bampton
8e538028b2
Enable Flake8 rule W391 (#6606)
Best practice is to have one blank line at the end of Python files.

Remove unneeded blank lines from the end of files
2022-11-13 11:00:07 +01:00
Step Security Bot
de8aae1619
[StepSecurity] ci: Harden GitHub Actions in merge-conflict-checker.yml (#6802) 2022-10-08 11:40:43 +05:30
Rohit Yadav
4d41b6bc44 Merge remote-tracking branch 'origin/4.17' 2022-08-09 12:33:39 +02:00
Rohit Yadav
2d0a2e388d .github: add codecov coverage check
It seems codecov can overcome the pull_request_target limitation/issue
compared to sonarcloud coverage. We've some other Apache projects who
use `codecov` now and this ticket to add support with infra:
https://issues.apache.org/jira/browse/INFRA-23561

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-08-09 12:29:19 +02:00
John Bampton
a039250b3b
.github: Lint modified files only with Super-Linter (#6531)
https://github.com/github/super-linter#standard-image
2022-07-07 11:57:29 +05:30
Rohit Yadav
711d4b9f0f Merge remote-tracking branch 'origin/4.17' 2022-07-01 11:26:10 +05:30
Rohit Yadav
85df82c2a9 Revert ".github: try using just sonarcloud job on pull_request target only"
This reverts commit aad73322aff1a5b1393c024949c6ba0b021a4c90.
2022-07-01 11:25:27 +05:30
neilnaveen
15ab481928
chore: Set permissions for GitHub actions (#6467)
Restrict the GitHub token permissions only to the required ones; this way, even if the attackers will succeed in compromising your workflow, they won’t be able to do much.

- Included permissions for the action. https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions

https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs

[Keeping your GitHub Actions and workflows secure Part 1: Preventing pwn requests](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)

Signed-off-by: neilnaveen <42328488+neilnaveen@users.noreply.github.com>
2022-07-01 11:24:04 +05:30
Rohit Yadav
5fe095ee0a Merge remote-tracking branch 'origin/4.17' 2022-06-30 15:28:55 +05:30
Rohit Yadav
aad73322af .github: try using just sonarcloud job on pull_request target only
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-06-30 15:27:20 +05:30
John Bampton
7ba43e447c
Enable flake8 rule W292 No newline at end of file (#6274)
https://www.flake8rules.com/rules/W292.html
2022-06-30 12:08:27 +05:30
Rohit Yadav
470de7fb54 .github: run coverage for pushes to main branch
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-05-04 02:23:47 +05:30
Rohit Yadav
17afe0510e .github: add support for PR decoration
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-05-04 01:02:38 +05:30
Rohit Yadav
2326b5e22a maven: update sonarcloud project
https://sonarcloud.io/project/configuration?id=apache_cloudstack

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-05-03 18:55:20 +05:30
Rohit Yadav
c379430dab .github: sonarcloud integration fixes
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-27 18:15:27 +05:30
Rohit Yadav
a2abd7fdc2 .github: fix typo in passing merge branch name
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-26 16:40:09 +05:30
Rohit Yadav
05e4f9515e .github: fix sonarcloud integration
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-26 16:35:07 +05:30
Rohit Yadav
9f27c992da .github: skip coverage data when string is empty
This fixes issue of empty rows in the PR coverage data

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-26 15:06:06 +05:30
Rohit Yadav
7233c38567 .github: enable PR sonar integration
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-26 14:43:23 +05:30
Rohit Yadav
40193a04eb .github: fix coverage issue, don't kick pkging automatically
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-20 16:37:32 +05:30
Rohit Yadav
15928f8cc8 .github: detect and kick packaging jobs
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-18 17:04:44 +05:30