4668 Commits

Author SHA1 Message Date
Daan Hoogland
fadb39ece7 Merge release branch 4.20 to main
* 4.20:
  merge errors fixed
  Restrict the migration of volumes attached to VMs in Starting state (#9725)
  server, plugin: enhance storage stats for IOPS (#10034)
  Introducing granular command timeouts global setting (#9659)
  Improve logging to include more identifiable information (#9873)
2025-01-08 14:01:19 +01:00
Abhishek Kumar
bd488c4bba
server, plugin: enhance storage stats for IOPS (#10034)
Adds framework layer change to allow retrieving and storing IOPS stats for storage pools. Custom `PrimaryStoreDriver` can implement method - `getStorageIopsStats` for returning IOPS stats. Existing method `getUsedIops` can also be overridden by such plugins when only used IOPS is returned.
For testing purpose, implementation has been added for simulator hypervisor plugin to return capacity and used IOPS for a pool.
For local storage pool, implementation has been added using iostat to return currently used IOPS.
StoragePoolResponse class has been updated to return IOPS values which allows showing IOPS values in UI for different storage pool related views and APIs.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-01-07 17:17:12 +05:30
Vishesh
a4224e58cc
Improve logging to include more identifiable information (#9873)
* Improve logging to include more identifiable information for kvm plugin

* Update logging for scaleio plugin

* Improve logging to include more identifiable information for default volume storage plugin

* Improve logging to include more identifiable information for agent managers

* Improve logging to include more identifiable information for Listeners

* Replace ids with objects or uuids


* Improve logging to include more identifiable information for engine

* Improve logging to include more identifiable information for server

* Fixups in engine

* Improve logging to include more identifiable information for plugins

* Improve logging to include more identifiable information for Cmd classes

* Fix toString method for StorageFilterTO.java
2025-01-06 16:42:37 +05:30
Daan Hoogland
9295a1624d Merge release branch 4.20 to main
* 4.20:
  VR: apply iptables rules when add/remove static routes (#10064)
  Certificate and VM hostname validation improvements (#10051)
  set ulimit for server according to redhat spec (#10040)
  kvm-storage: provide isVMMigrate information to storage plugins (#10093)
  Allow config drive deletion of migrated VM, on host maintenance (#10045)
  linstor: improve heartbeat check with also asking linstor (#10105)
  server: simplify role change validation (#9173)
  UI: create VPC network offering with conserve mode (#10082)
  server: fix typo removeaccessvpn in VirtualRouterElement (#10086)
  UI: remove duplicated Instance Name in Public IP details page (#10087)
  UI: Fixes in the Usage UI (#10000)
  SAML2: add cookie with HttpOnly too #10013 (#10047)
  ui: Allow font-awesome icon usage and optimise icon size inconsistency (#9744)
2024-12-20 14:37:49 +01:00
Daan Hoogland
b7f0aac519 Merge branch '4.19' into 4.20 2024-12-20 14:34:39 +01:00
Bernardo De Marco Gonçalves
73c3339bf1
List users by their authentication source (#10115) 2024-12-19 14:12:55 +01:00
Suresh Kumar Anaparti
b4ad04badf
Allow config drive deletion of migrated VM, on host maintenance (#10045) 2024-12-18 09:12:28 +01:00
Bernardo De Marco Gonçalves
f75a194c09
Persist IP addresses related to VM access via CPVM (#9534) 2024-12-10 11:43:17 +01:00
Daan Hoogland
205ebfb8b5 Merge release branch 4.20 to main
* 4.20:
  UI: Fix userdata and load balancer selection (#10016)
  Prevent password updates for SAML and LDAP users (#9999)
  cloudstack-migrate-databases: sql AND added (#10033)
  engine/schema: move SQLs to 4.20.0 to 4.20.1 upgrade (#10018)
  Remove user from project before deletion (#10008)
  Simplify validation for creating volume templates via UI (#9828)
2024-12-04 13:05:32 +01:00
Wei Zhou
b3dc402aa8
engine/schema: move SQLs to 4.20.0 to 4.20.1 upgrade (#10018) 2024-12-04 08:11:28 +01:00
Daan Hoogland
9b6f9b5f7d Merge release branch 4.20 to main
* 4.20:
  UI: Tooltip on the host information card to display the CPU speed in MHz and the memory value in MB (to 3 decimal places) (#9971)
  UI: Allow accounts of the `User` type to add other accounts or users to projects through UI (#9927)
  enable to create VPC portfowarding rules with source cidr (#7081)
  Add new column `last_id` to the table volumes (#9759)
  Allow VMWare import via another host (#9787)
  Linstor: add support for ISO block devices and direct download (#9792)
  get expunged VM data for job result (#9949)
  fix section divider display on auth page (#9966)
2024-12-03 16:33:51 +01:00
Daan Hoogland
da54234585 Merge branch '4.19' into 4.20.merge 2024-12-03 16:32:15 +01:00
Wei Zhou
da94ae2c1c
Merge remote-tracking branch 'apache/4.20' 2024-12-03 09:44:42 +01:00
Vishesh
52e7b41a54
Add support for functionality in hypervisor class (#9736)
* Add support for functionality in hypervisor class

* Address comments

* address comments
2024-12-03 12:20:59 +05:30
Abhisar Sinha
d17de834a5
Disable API Key Access for users, accounts and domains (#9741)
* cli changes to update user/account, list by apikeyaccess, domain level setting

* UI changes for updating user/account and searchfilter in listview

* make the api parameters and setting accessible only to root admin

* revert changes to ui/package-lock.json

* minor changes to description strings

* UT for ApiServer and AccountManagerImpl classes

* fix pre-commit failure

* Added a constant for the string System

* UT for searchForUsers and searchForAccounts

* Fix marvin test error

* Update schema to use idempotent add column

* Fix `updateTemplatePermission` when the UI is set to a language other than English (#9766)

* Fix updateTemplatePermission UI in non-english language

* Improve fix

---------

Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>

* Added user name uuid to logging

* Add events when api key access is changed via api or config setting

* fix the userid for api key access update event

* Fix ut failure after event logging

* Convert drop down to radio-button in edit user and account

* Add ApiKeyAccess status in User InfoCard for Users if Api key is generated

* Return apiKeyAccess in user and account response only for Root Admin

* fixed noredist build failure

* Show apikeyaccess on the left panel in the user view for root admins as well

* don't show divider if apiKeyAccess is not shown to user

* Fix events generated to set Username, Account and Domain of the caller correctly

* cli changes to update user/account, list by apikeyaccess, domain level setting

* UI changes for updating user/account and searchfilter in listview

* make the api parameters and setting accessible only to root admin

* revert changes to ui/package-lock.json

* minor changes to description strings

* UT for ApiServer and AccountManagerImpl classes

* fix pre-commit failure

* Added a constant for the string System

* UT for searchForUsers and searchForAccounts

* Fix marvin test error

* Update schema to use idempotent add column

* Added user name uuid to logging

* Add events when api key access is changed via api or config setting

* fix the userid for api key access update event

* Fix ut failure after event logging

* Convert drop down to radio-button in edit user and account

* Add ApiKeyAccess status in User InfoCard for Users if Api key is generated

* Return apiKeyAccess in user and account response only for Root Admin

* fixed noredist build failure

* Show apikeyaccess on the left panel in the user view for root admins as well

* don't show divider if apiKeyAccess is not shown to user

* Fix events generated to set Username, Account and Domain of the caller correctly

* Added DB upgrade path from 42000 to 42010

---------

Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Lucas Martins <56271185+lucas-a-martins@users.noreply.github.com>
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
2024-12-03 12:10:54 +05:30
Wei Zhou
8a1da3804c
Resize volume: add pool capacity disablethreshold for resize and allow volume auto migration (#9761)
* server: add global settings for volume resize

* resizeVolume: support automigrate

* Address Suresh's comments

* Update api/src/main/java/org/apache/cloudstack/api/command/user/volume/ResizeVolumeCmd.java

Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>

* address Suresh's comments

* UI: add autoMigrate to resizeVolume

* resizevolume: add unit tests

* resizevolume: add unit test for Allocated volume

---------

Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2024-12-02 10:28:14 +05:30
Wei Zhou
34056d956c
Improvement: management server peer states (#9885)
* Improvement: management server peer states

* Update pr9885: consider new mgmt server node which has msId=managementServerNodeId

* Update pr9885: update global config description

* Update pr9885: update label on UI

* framework: Do not update mshost_peer when mgmt server is Up as it will be updated by status update

* mgmt: Update state to Up when mgmt server writes heartbeat to db

* mgmt: change Service IP to Management IP

---------

Co-authored-by: Boris Stoyanov - a.k.a Bobby <bss.stoyanov@gmail.com>
2024-12-02 10:26:20 +05:30
Wei Zhou
22a6604491
Routed mode: minor improvements and fixes (#9697)
* Routed: add global setting to enable/disable routed mode

* Routed: add unit tests

* Routed: add unit tests RoutedIpv4ManagerImplTest

* PR9697: fix end of files lint error

* PR9697: fix unit tests error with JDK17

* Routed: do not show ROUTED mode when create network/vpc offering if routed is disabled

* Routed: check if routed network is enabled when create ipv4 subnet/bgp peer/ASN range/network/vpc

* Routed: UI bug fix and improvements

* Routed: apply available BGP peers when VR is restarted
2024-11-30 08:45:15 +01:00
Rodrigo D. Lopez
4189bac8e0
enable to create VPC portfowarding rules with source cidr (#7081)
Co-authored-by: Lopez <rodrigo@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2024-11-28 17:53:07 +01:00
Vishesh
37eec06166
Allow VMWare import via another host (#9787)
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-11-28 17:50:08 +01:00
João Jandre
d9774a8462 Updating pom.xml version numbers for release 4.21.0.0-SNAPSHOT
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-27 11:47:06 -03:00
João Jandre
c63c7ee63e Updating pom.xml version numbers for release 4.20.1.0-SNAPSHOT
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-27 11:40:45 -03:00
João Jandre
2fe3fcef7c Updating pom.xml version numbers for release 4.20.0.0
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-19 08:54:07 -03:00
Pearl Dsilva
843c1950e7
Fix Listing backup schedules (#9749) 2024-10-15 09:12:10 +02:00
Daan Hoogland
dbfc7f23a7 Merge branch '4.19' 2024-10-11 17:59:46 +02:00
Daan Hoogland
7d70e32378 fix quota resource access validation 2024-10-11 17:24:05 +02:00
Vishesh
21d107c349
Merge branch '4.19' 2024-09-24 14:04:51 +05:30
Vishesh
9df783ca4b
Filter out networks without access while getting networks with SG with free IPs (#9596) 2024-09-20 16:43:54 +02:00
Abhisar Sinha
a88967bf61
Minor naming changes in Shared FileSystems 4.20 Feature (#9675) 2024-09-17 08:53:50 +02:00
Vishesh
0655075f51
Feature: Forgot password (#9509)
* Feature: Forgot password

* Address comments

* fixups

* Make forgot password disabled by default

* Apply suggestions from code review

* Address comments
2024-09-10 12:55:28 -03:00
Bernardo De Marco Gonçalves
6ec3c486dd
Enhance the listAffinityGroups API by adding the dedicated resources related to an affinity group (#9188)
* add dedicated resource response

* populate dedicatedresources field

* change affinity group name and description when it contains dedicated resources

* display dedicatedresources on UI

* add end of line to DedicatedResourceResponse class

* remove unnecessary fully qualified names
2024-09-10 08:12:41 -03:00
Vishesh
1303a4f323
Feature: Allow adding delete protection for VMs & volumes (#9633)
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-09-09 14:44:50 +02:00
Pearl Dsilva
f8d8a9c7b3
NSX Integration fixes (#8906)
* Prevent addition of duplicate PF rules on scale up and no rules left behind on scale down (#32)

* fix missing dependency injection

* NSX: Fix concurrency issues on port forwarding rules deletion (#37)

* Fix concurrency issues on port forwarding rules deletion

* Refactor objectExists

* Fix unit test

* Fix test

* Small fixes

* CKS: Externalize control and worker node setup wait time and installation attempts (#38)

* NSX: Add shared network support (#41)

* NSX: Fix number of physical networks for Guest traffic checks and leftover rules on CKS cluster deletion (#45)

* Fix pf rules removal on CKS cluster deletion

* Fix check for number of physical networks for guest traffic

* Fix unit test

* fix logger

* NSX: Handle CheckHealthCommand to avoid host disconnection and errors on APIs

* NSX: Handle CheckHealthCommand to avoid host disconnection and errors on APIs

* Remove unused string

* fix logger

* Update UDP active monitor to ICMP

* Fix NPE on restarting VPC with additional public IPs

* NSX / VPC: Reuse Source NAT IP from systemVM range on restarts

* CKS: Public IP not found for VPC networks

* Externalize retries and inverval for NSX segment deletion (#67)

* remove unused import

* remove duplicate imports

* remove unused import

* revert externalizing cks settings

* fix test

* Refactor log messages

* Address comments

* Fix issue caused due to forward merge: 90fe1d

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-06 16:56:50 -03:00
Nicolas Vazquez
8c8d115a1e
feature: Support Multi-arch Zones (#9619)
This introduces the multi-arch zones, allowing users to select the VM arch upon deployment. 

Multi-arch zone support in CloudStack can allow admins to mix x86_64 & arm64 hosts within the same zone with the following changes proposed:
- All hosts in a clusters need to be homogenous, wrt host CPU type (amd64 vs arm64) and hypevisor
- Arch-aware templates & ISOs:
   -  Add support for a new arch field (default set of: amd64 and arm64), when unspecified defaults to amd64 and for existing templates & iso
   -  Allow admins to edit the arch type of the registered template & iso
- Arch-aware clusters and host:
   - Add new attribute field for cluster and hosts (kvm host agents can automatically report this, arch of the first host of the cluster is cluster's architecture), defaults to amd64 when not specified
   - Allow admins to edit the arch of an existing cluster
- VM deployment form (UI):
   - In a multi-arch zone/env, the VM deployment form can allow some kind of template/iso filtration in the UI
   - Users should be able to select arch: amd64 & arm64; but this is shown only in a multi-arch zone (env)
- VM orchestration and lifecycle operations:
   - Use of VM/template's arch to correctly decide where to provision the VM (on the correct strictly arch-matching host/clusters) & other lifecycle operations (such as migration from/to arch-matching hosts)

Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-06 12:14:54 +05:30
Wei Zhou
679ce1a639
feature: Dynamic and Static Routing (#9470)
This PR contains 3 features

- IPv4 Static Routing (Routed mode) #9346
Design document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=306153967

- AS Numbers Management #9410
Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/BGP+AS+Numbers+Management


- Dynamic routing
Design Document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=315492858

- Document: https://github.com/apache/cloudstack-documentation/pull/419

Rename nsx mode to routing mode

by
```
git grep -l nsx_mode  |xargs sed -i "s/nsx_mode/routing_mode/g"
git grep -l nsxmode  |xargs sed -i "s/nsxmode/routingmode/g"
git grep -l nsxMode  |xargs sed -i "s/nsxMode/routingMode/g"
git grep -l NsxMode  |xargs sed -i "s/NsxMode/RoutingMode/g"
```
- re-organize sql changes
- fix NPE as rules do not have public ip
- fix missing destination cidr in ingress rules
- disable network usage for routed network
- fix DB exception as network_id is -1 during network creation
- apply ingress/egress routing rules
- VR changes to configure nft rules for isolated network
- VR: setup nft rule for control network
- VR: flush all iptables rules
- fix NPE which is because ingress rules do not have public ip associated
- fix dest cidr is missing in nft tables
- add ip4 routing and ip4 routes to list network and list vpc response
- fix ingress rule is missing when vr is restarted
- fix icmp types in nft rules
- add tab to manage routing firewall rules
- fix ingress rules are not applied when VR is restarted
- add default rules in FORWARD chain
- fix create vpc offerings
- fix public ip is not assigned to vpc
- fix network offering is not listed when create vpc tier
- add is_routing to boot args of vpc vr
- remove table ip4_firewall in vpc vr
- release or remove subnet when remove a network
- implemenent fw_vpcrouter_routing
- fix wrong ip familty when flush ipv4 rules
- fix acl rules are not applied due to wrong version (should be 6 which means ip6 rules are removed)
- add default rules for vpc tiers so that tcp connections (e.g. ssh) work
- append policy rules after default rules
- remove /usr/local/cloud/systemvm/ in routers
- throw an exception when allocate subnet with cidrsize
- fix some TODOs
- add new parameters to update API
- return type Ipv4GuestSubnetNetworkMap when get or create subnet
- fix firewall rules are broken
- add domain_id and account_id to db
- add domain/account/project to ipv4 subnet response
- create ipv4 subnet for domain/account/project
- check conflict when update ipv4 subnet
- ui changes
- add parent subnet to response
- add list for ipv4 subnet
- implement some methods
- fix list subnets for guest networks by zoneid
- UI changes
- fix delete ipv4 subnet for network
- fix ipv4 subnet is set to zone guest network cidr if cidrsize is specified
- add zone info to response if parent subnet is null but network is not
- fix gateway/cidr is not set when create network with cidrsize
- fix order of nft rules in the VRs

* Routed v24

- add classes in marvin base.py

* Routed v25

- add test_01_subnet_zone
- fix dedicate to domain/account failure
- list subnets for network by keyword and subnet

* Routed v26: implement subnet auto-allocation

- add utils for split ip ranges into small subnets
- add utils to get start/end ip of a cidr
- implement subnet auto-generation
- add global settings

* Routed 27: add subnet for VPC

- add db column for vpc_id
- add db record for vpc
- remove db record when delete a vpc
- add checkConflicts methods
- remove duplicated settings
- check ipv4 cidr when create subnet

* Routed v28: update smoke tests

- update test_ipv4_routing.py
- search subnets by networkid

* Routed 29: fix vpc and add more tests

- fix createnetwork in vpc
- add vpc id/name to response
- fix zone id/name are not displayed in some cases
- add smoke test for vpc
- add smoke tests for failed cases
- add smoke test for connectivity checks
- marvin: add "-q" to ssh command

* Routed 31: ui and smoke tests

- UI: add link to network in list view
- add nftables rules check in VRs

* Routed 32: add chain OUTPUT and more rules

- fix the issue 80/443/8080 is not reachable from VR itself

```
2024-06-27 10:21:52,121 INFO     Executing: systemctl start cloud-password-server@172.31.1.1
2024-06-27 10:21:52,128 INFO     Service cloud-password-server@172.31.1.1 start
2024-06-27 10:21:52,129 INFO     Executing: ps aux
2024-06-27 10:24:02,175 ERROR    Failed to update password server due to: <urlopen error [Errno 110] Connection timed out>
```

* Routed: fix dns search from VMs in Isolated networks

* Routed: fix VPC dns issue due to gateway IP is missing in cloud.conf

This is caused by NSX integration, and fixed by
https://github.com/apache/cloudstack/pull/9102/

* Routed: rename routing_mode to network_mode

* Routed: replace centos5.5 template in smoke test as dhclient does not work in the vms

// this does not work
refer to https://dominikrys.com/posts/disable-udp-checksum-validation/#ignoring-udp-checksums-with-nftables
and
https://forum.openwrt.org/t/udp-checksum-with-nftables/161522/11

the vm should have checksum offloading disabled

* Routed: fix smoke test due to wrong cidrlist of egress rules and missing ingress rule from VR

* PR 9346: fix lint error schema-41910to42000.sql

* PR 9346: ui polish v1

* PR 9346: create VPC with cidrsize

* Routed: fix test failures with test_network_ipv6 and test_vpc_ipv6 due to 'ssh -q'

* Routed: fix /usr/local/cloud/systemvm/ are removed after SSVM/CPVM reboot

* Routed: fix IP of additional nics of VPC VR is not gateway

* PR 9346: fix cidrsize check when create VPC with cidrsize

* Routed: fix test/integration/smoke/test_ipv4_routing.py:279:16: E713 test for membership should be 'not in'

* PR9346: fix/Update api

* PR 9346: set response object name

* PR9346: UI refactor and small fixes

* PR9346: change return type of getNetworkMode

* PR9346: move IPv4 subnet to seperated tab

* PR9346: revert IpRangesTabGuest.vue back to original

* PR9346: fix remove ipv4 subnet on UI

* PR9346: fix test_ipv4_routing.py

* AS Number Range Management

* Create AS Number Range for a Zone

* Fix build

* Add ListASNRange and fix create ASN range

* Add List AS numbers

* Add UI for AS Numbers

* Fix UI and filter AS Numbers

* Add AS Number on Isolated network creation and refactor UI and response

* Release AS Number

* Add network offering new columns

* Add UI support to view and add AS number and configure network offering

* Automatically assign AS Number if not specify AS number

* update variable name

* Fix routing mode check

* UI: Only allow selecting AS number when routing mode is Dynamic and specifyAsNumber is true

* UI: Only pass AS number when supported by the network offering

* Release AS number on network deletion

* Add deleteASNRange command (#81)

* API: List ASNumbers by asnumber (#83)

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* AS number management extensions

* Support AS number on VPC tier creation based on the offering

* Fix delete AS Range

* Fix UI values

* UI: Minor fix for releasing AS number

* UI: Move management of AS Range to Zone details view

* Fix specify_as_number column in network_offering table to set the default false

* Add events for AS number operations

* Allow users to list AS Numbers and fix network form for Normal users

* Add AS number details to list networks response

* Fix Allocated time format

* Fix Allocated time format

* support in details view too

* Fix: Do not release AS number if acquired network requires AS number

* Fix: Do not release AS number if acquired network requires AS number

* Fix typo

* Fix allocated release

* Fix event type

* UI: Add Routing mode and Specify AS to the network offering details

* UI: Add Routing mode and Specify AS to the network offering details

* Address comment

* Fix release AS number of network deletion

* Fix release AS number of network deletion

* Fix

* Restore release to its place based on the boolean

* Rename boolean

* API: Add networkId as listASNumber parameter

* Add Network name to the search view filter for AS numbers

* Present allocated time in human readable format - Pubilc IP / AS Numbers

* Add account / domain filter for AS numbers

* Add support for AS numbers on VPC offerings

* Refactor AS number allocation to VPC and non VPC isolated networks

* Checkstyle

* Add support for AS numbers on VPC offerings

* extend vpc offering view and vpcoffering response

* merge https://github.com/shapeblue/cloudstack-playtika/pull/115 and change network_id of as_numbers to include vpc_id

* Display AS number of VPC tiers as the AS number of the VPC

* extend asnumber response and ui support

* improve UI and as number response to view VPC details

* List only dynamic offerings for vpc tiers with specify as numbers

* Fix release AS number

* Fix AS number displayed as 0 when no AS number assigned

* Fix VPC offering creation without specify AS

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix release AS number on VPC deletion

* Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java

* Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java

* Fix missing column on asnumber table

* Fix listASNumbers API to support vpcid and obtain AS number from vpc for tiers

* Prevent listing 0 AS number for VPC

* Fix create Isolated Network form

* Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

* Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

* Dynamic: move routingmode/specifyasn after networkmode in AddNetworkOffering.vue on UI

* Dynamic: fix ip4routing in network response

* Dynamic/systemvm: add FRR to systemvm template

* Dynamic: BGP peers (DB,VO,Dao)

* Dynamic: BGP peers (VR/server)

* Dynamic: v3

- remove BgpPeer class
- fix vpc vr has bgp peers of only 1 tier
- rename ip4_cidr to guest_ip4_cidr
- rename ip6_cidr to guest_ip6_cidr
- generate /etc/frr/frr.conf
- apply BGP peers on Dynamic-Routed network even if there is no BGP peers

* Dynamic v4: fix vpc vr

- fix duplicated guest cidr in frr.conf in vpc vr

todo
- restart frr / reload frr (reload will cause bgp session to Policy state)
- apis for bgp peers
- assign/release bgp peer from/to network

* Dynamic v5: add apis for bgp peers

* Dynamic v6: fix bugs

- set response object name
- remove required as number when update
- fix checks when update
- allow regular users to list bgp peers

* Dynamic v7: move apis to bgp sub-dir

* Dynamic v8: add tab for manage BGP peers on UI

* Dynamic v9: fix update bgp with same config

* Dynamiv v10: add changeBgpPeersForNetworkCmd

* Dynamic v11: create network with bgppeerids

- create network with bgppeerids
- add marvin classes
- add smoke tests
- remove uuid from bgp_peer_network_map
- fix created/removed in bgp_peer_network_map
- remove bgppeers when remove a network
- UI: fix delete bgp peer

* Dynamic v12: add test for vpc tiers

* Dynamic v13: bug fixes

- fix change BGP peers for network in Allocated state
- fix listing network returns removed record
- fix all vpc tiers have the same settings
- remove BGP peers as part of network removal
- remove FRR settings for vpc tiers without any BGP peers
- UI: fix no error msg when change BGP peers

* Dynamic v14: assign BGP Peers for VPC instead of VPC tiers

- create vpc with bgppeerids
- do not allow create/update vpc tier with bgppeerids
- apply all bgp peers when create/delete a vpc tier
- UI: change bgp peers for vpc
- test: update tests on vpc

* Dynamic: fix build errors after merging as number PR

* Dynamic: fix TODOs

* Dynamic: fix smoke test on VPC

* Allow creation of networks by users with as numbers

* Address review comments

* Move BGPService to bgp package and inject it on BaseCmd

* Revert changes for CKS and address more comments

* Display left side menu option for AS number only for root admin

* Dynamic: create/update BGP peer with details

refer to https://docs.frrouting.org/en/latest/bgp.html

* Dynamic: fix build error and remove access to ListBgpPeers cmd for regular users

* Dynamic: assign all zone BGP peers to user networks

* Dynamic: show BGP peer info of networks only for root admin

* AS number: disable specifyasnumber for non-NSX offerings

* Dynamic: pass bgppeer details to command and fix typo with ip6 addr

* Dynamic: list BGP peers by isdedicated, and fix change bgppeers for network/vpc

* Dynamic: add UI labels

* Dynamic: add bgp peers to vpc response

* Dynamic: list bgp peers by keyword, fix list by asnumber

* Dynamic: fix list bgppeers by keyword and db schema

* Dynamic: fix list bgppeers do not return dedicated peers

* Dynamic: update UI when create network/vpc offering

* Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* Update tools/marvin/setup.py

* Dynamic: network mode must be same when update a network with new offering

* Dynamic: add method networkModel.isAnyServiceSupportedInNetwork

* Dynamic: rename APIs and classes

* Dynamic: fix unit tests due to previous changes

* Dynamic: validateNetworkCidrSize when auto-create subnet

* Dynamic: check AS number overlap

* Dynamic: add ActionEvent

* Dynamic: small code optimization

* Dynamic: fix ui bugs after api rename

* Dynamic: add marvin and test for ASN ranges and AS numbers

* Dynamic: add account setting use.system.bgp.peers

also
- change the default value of routed.ipv4.vpc.max.cidr.size and routed.ipv4.vpc.min.cidr.size
- change the category of settings

* static: fix ui error when delete zone ipv4 subnets

* static: small UI polish

* Dynamic: throw exception when as number is required but not passed

* Dynamic: fix typo when create FRR directory which causes network deletion failures

* Dynamic: connect to ALL (or ALL dedicated) BGP peers if no BGP peer mapping for the network/vpc

* Dynamic: throw exception when as number is required for VPC but not passed

* Dynamic: list bgp peers by useSystemBgpPeers

* Dynamic: fix frr config in VPC VR when change bgp peers

* Dynamic: create frr config even if there is no VPC tiers

* Dynamic: list bgp peers by zoneid (required for account) and account

* Dynamic: only apply FRR config for vpc tiers with dynamic routing

* Dynamic: donot send commands to router if commands size is 0

* Dynamic: fix 'new IPv6 address is not valid' when update bgp peer without IPv6

* Dynamic: throw exception if fail to allocate AS number when create network/vpc with dynamic routing

* Dynamic: enable ipv6 unicast and 'ip nht resolve-via-default'

* Dynamic: delete network/vpc if fail to allocate AS number when create network/vpc with dynamic routing

* test: add unit tests for ASN APIs

* test: add unit tests for core module

* test: add unit tests for API responses

* test: add unit tests for BgpPeerTO

* test: add minor changes

* test: add tests for create/delete/update/list RoutingFirewallRuleCmd

* Static: show ip4 routes for vpc tiers

* test: fix smoke test failure caused by type change of as number

* test: add test for Ipv4SubnetForZoneCmd

* test: add test for Ipv4SubnetForGuestNetworkCmd and BgpPeerCmd

* UI: do not show redundant router when network mode is ROUTED as RVR is not supported

* UI: hide 'Conserve mode' when networkmode is ROUTED

* test: add unit tests for ListASNumbersCmdTest

* Static: remove allocated IPv4 subnet when delete a network or vpc

* test: add unit tests for BgpPeersRules

* Dynamic: set ipv4routing from network offering

* server: list as numbers and ipv4 subnets by keyword

* server: remove dedicated bgp peers and ipv4 subnets when delete an account or domain

* server: fix dedicated ipv4 subnet is allocated to other accounts

* UI: fix allocated time format

* server: ignore project is projectid is -1 so bgppeers/ipv4subnets works in project view

* UI: add project column to bgp peers and ipv4 subnets

* server: fix list AS numbers by domain admin or normal user

* server: fix network creation when ipv4 subnet is dedicated

* UI: polish network.js

* Dynamic: fix frr config for ipv6 routing

* Static routing: support cks cluster

* Static: get/create IPv4 subnet from dedicated subnets at first

* Dynamic: add BGP peers tab

* Static: remove redundant loops

* api: add since to api and response

* server: add unit tests

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-06 08:55:17 +05:30
Rohit Yadav
85765c3125
backup: simple NAS backup plugin for KVM (#9451)
This is a simple NAS backup plugin for KVM which may be later expanded for other hypervisors. This backup plugin aims to use shared NAS storage on KVM hosts such as NFS (or CephFS and others in future), which is used to backup fully cloned VMs for backup & restore operations. This may NOT be as efficient and performant as some of the other B&R providers, but maybe useful for some KVM environments who are okay to only have full-instance backups and limited functionality.

Design & Implementation follows the `networker` B&R plugin, which is simply:

- Implement B&R plugin interfaces
- Use cmd-answer pattern to execute backup and restore operations on KVM host when VM is running (or needs to be restored) - instead of a B&R API client, relies on answers from KVM agent which executes the operations
- Backups are full VM domain snapshots, copied to a VM-specific folders on a NAS target (NFS) along with a domain XML
- Backup uses libvirt feature: https://libvirt.org/kbase/live_full_disk_backup.html orchestrated via virsh/bash script (nasbackup.sh) as the libvirt-java lacks the bindings
- Supported instance volume storage for restore operations: NFS & local storage

Refer the doc PR for feature limitations and usage details:
https://github.com/apache/cloudstack-documentation/pull/429

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-09-05 22:19:13 +05:30
Wido den Hollander
c3f0d14d31
storage/object: Add support for Ceph RGW Object Store (#8389)
This feature adds support for Ceph's RADOS Gateway (RGW) support for the
Object Store feature of CloudStack.

The RGW of Ceph is Amazon S3 compliant and is therefor an easy and straigforward
implementation of basic S3 features.

Existing Ceph environments can have the RGW added as an additional feature to a
cluster already providing RBD (Block Device) to a CloudStack environment.

Introduce the BucketTO to pass to the drivers. This replaces just passing the bucket's name.

Some upcoming drivers require more information then just the bucket name to perform their actions,
for example they require the access and secret key which belong to the account of this bucket.

This is leftover code from a long time ago and this validation test has nu influence
on the end result on how a URL will be used afterwards.

We should support hosts pointing to an IPv6(-only) address out of the box.

For the code it does not matter if it's IPv4 or IPv6. This is the admin's choice.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-05 17:58:39 +05:30
Abhisar Sinha
605534b417
feature: Shared Storage Filesystem as a First Class Feature (#9208)
This PR implements Storage filesystem as a first class feature.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Filesystem+as+a+First+Class+Feature

Documentation PR: apache/cloudstack-documentation#420

Co-authored-by: Wei Zhou <weizhou@apache.org>
2024-09-05 17:22:32 +05:30
Vishesh
72d0546d8b
Shared Network Firewall (Security groups) in Advanced zone without security groups (#9415) 2024-09-05 10:35:05 +02:00
Harikrishna
716ab205ea
Added update, enable, disable events to the updateStoragePool API (#9543) 2024-09-05 08:39:06 +02:00
Abhisar Sinha
b7a2b0419c
server: Global setting to allow/disallow users to force stop a vm (#9569)
Global setting to allow/disallow users to force stop a vm

Fixes #6629

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Co-authored-by: Rohit Yadav <rohityadav89@gmail.com>
Co-authored-by: Bernardo De Marco Gonçalves <bernardomg2004@gmail.com>
2024-09-05 09:46:45 +05:30
Nicolas Vazquez
36d37f70a8
Display associated resource name on storage pools objects (#9449) 2024-09-05 09:39:07 +05:30
Rohit Yadav
60c9c9adb6 Merge remote-tracking branch 'origin/4.19'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-04 13:20:42 +05:30
Suresh Kumar Anaparti
b78aede2b7
Updated listStoragePools response - added new managed parameter (#9588) 2024-09-04 11:54:33 +05:30
Abhishek Kumar
5a496e725b Merge remote-tracking branch 'apache/4.19' 2024-08-28 16:07:31 +05:30
Suresh Kumar Anaparti
c05edc40f4
Updated invalid parameter/value to com.cloud.exception.InvalidParameterValueException (#9455) 2024-08-28 08:54:43 +02:00
Lucas Martins
70131be8c2
Fix deleteAccount API to prevent deletion of the caller (#8743)
* Fix API deleteAccount deleting caller's account

* Remove extra line

* Add unit tests and change message

* apply suggestion

Co-authored-by: Bernardo De Marco Gonçalves <bernardomg2004@gmail.com>

* remove extra parentheses

Co-authored-by: Bernardo De Marco Gonçalves <bernardomg2004@gmail.com>

* Update server/src/test/java/com/cloud/user/AccountManagerImplTest.java

Co-authored-by: Bernardo De Marco Gonçalves <bernardomg2004@gmail.com>

---------

Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
Co-authored-by: Bernardo De Marco Gonçalves <bernardomg2004@gmail.com>
2024-08-27 16:13:30 -03:00
Nicolas Vazquez
d32ace6731
New Feature: Enable/Disable Roles (#9549)
* New feature: Enable/Disable Roles

* Fixes

* Fix unit tests
2024-08-27 14:32:07 -03:00
Suresh Kumar Anaparti
674129cd58
Update project account for all the events with project account owner, except for create project event (#9572) 2024-08-27 12:37:28 +02:00
Wei Zhou
f5efe56fb5
Merge remote-tracking branch 'apache/4.19' 2024-08-26 17:34:15 +02:00