296 Commits

Author SHA1 Message Date
Wei Zhou
22bf8fd5d1
Merge remote-tracking branch 'apache/4.20' 2025-09-22 16:31:01 +02:00
Wei Zhou
393b5d2b77
VR: consider NICs for remote access VPN when apply dhcp entry (#11681) 2025-09-22 10:25:21 +02:00
dahn
aca8732102
[router] make a distinction between fatal errors, warnings and unknown as healthcheck result (#10710)
* [routers] distiction between fatal failure and warning or unknown on healthchecks

* UI status for router health checks

* status from scripts varied

* automation signalled errors

* revert removal of update sql

* upgradeversion

* move config item and further cleanup

* handling services better

* backwards compatible response

---------

Co-authored-by: Daan Hoogland <dahn@apache.org>
2025-09-22 11:39:05 +05:30
John Bampton
33cfedee46
pre-commit add hook check-shebang-scripts-are-executable for Shell (#10150) 2025-09-17 14:22:28 +02:00
Wei Zhou
70a4503ea1
Merge remote-tracking branch 'apache/4.20' 2025-09-11 14:04:52 +02:00
Wei Zhou
b46e29dc67
Improvement: SSL offloading with Virtual Router (#11468)
* SSL offloading with Virtual Router

* PR11468: fix pre-commit errors

* PR11468: api->getAPI/postAPI in UI

* SSL: add smoke tests for VPC in user project

* PR11468: address Daan's comments

* Fix test/integration/smoke/test_ssl_offloading.py

* SSL: remove ssl certificates when clean up account

* SSL offloading: add unit tests

* SSL offloading: UI fixes part 1

* SSL offloading: UI changes part 2

* SSL offloading: add more unit tests

* SSL offloading: more unit tests 3

* SSL offloading: wrong check

* SSL offloading: more and more unit tests

* SSL offloading: add testUpdateLoadBalancerRule5
2025-09-11 16:37:18 +05:30
Fabricio Duarte
5d32492676
IPv6 firewall: accept packets from related and established connections (#10970)
* IPv6 firewall: accept related packets and packets from established connections

* Remove rule from input chain
2025-09-10 16:37:17 +02:00
Wei Zhou
66a584624f
Update md5sum to sha512sum (#11134)
* Update md5sum to sha512sum

* Move SQL to schema-42100to42200.sql
2025-09-03 11:12:43 +02:00
Suresh Kumar Anaparti
2eb80e0361
Merge branch '4.20' 2025-08-29 15:36:16 +05:30
Suresh Kumar Anaparti
ca62a7dd50
Merge branch '4.19' into 4.20 2025-08-29 15:31:00 +05:30
Wei Zhou
05e7a257ca
VPC VR: return UNKNOWN redundant state if no guest nics (#11518)
* VPC VR: return UNKNOWN redundant state if no guest nics
2025-08-29 12:51:33 +05:30
Daan Hoogland
0b3959221b Merge branch '4.20' 2025-07-29 16:50:55 +02:00
Pearl Dsilva
890386e949
Allow custom NTP servers for CPVM (#11210) 2025-07-25 15:21:06 +02:00
Pearl Dsilva
0d4147f3f6
Netris Network Plugin Integration with CloudStack (#10458)
The Netris Plugin introduces Netris as a network service provider in CloudStack to be able to create and manage Virtual Private Clouds (VPCs) in CloudStack, being able to orchestrate the following network functionalities:

- Network segmentation with Netris-VXLAN isolation method
- Routing between "public" IP and network segments with an ACS ROUTED mode offering
- SourceNAT, DNAT, 1:1 NAT between "public" IP and network segments with an ACS NATTED mode offering
- Routing between VPC network segments (tiers in ACS nomenclature)
- Access Lists (ACLs) between VPC tiers and "public" network (TCP, UDP, ICMP) both as global egress rules and "public" IP specific ingress rules.
- ACLs between VPC network tiers (TCP, UDP, ICMP)
- External load balancing – between VPC network tiers and "public" IP
- Internal load balancing – between VPC network tiers
- CloudStack Virtual Router services (DHCP, DNS, UserData, Password Injection, etc…)
2025-07-25 15:26:42 +05:30
Nicolas Vazquez
6adfda2818
CKS Enhancements (#9102)
CKS Enhancements:

* Ability to specify different compute or service offerings for different types of CKS cluster nodes – worker, master or etcd

* Ability to use CKS ready custom templates for CKS cluster nodes

* Add and Remove external nodes to and from a kubernetes cluster

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Update remove node timeout global setting

* CKS/NSX : Missing variables in worker nodes

* CKS: Fix ISO attach logic

* CKS: Fix ISO attach logic

* address comment

* Fix Port - Node mapping when cluster is scaled in the presence of external node(s)

* CKS: Externalize control and worker node setup wait time and installation attempts

* Fix logger

* Add missing headers and fix end of line on files

* CKS Mark Nodes for Manual Upgrade and Filter Nodes to add to CKS cluster from the same network

* Add support to deploy CKS cluster nodes on hosts dedicated to a domain

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* Support unstacked ETCD

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix CKS cluster scaling and minor UI improvement

* Reuse k8s cluster public IP for etcd nodes and rename etcd nodes

* Fix DNS resolver issue

* Update UDP active monitor to ICMP

* Add hypervisor type to CKS cluster creation to fix CKS cluster creation when External hosts added

* Fix build

* Fix logger

* Modify hypervisor param description in the create CKS cluster API

* CKS delete fails when external nodes are present

* CKS delete fails when external nodes are present

* address comment

* Improve network rules cleanup on failure adding external nodes to CKS cluster

* UI: Fix etcd template was not honoured

* UI: Fix etcd template was not honoured

* Refactor

* CKS: Exclude etcd nodes when calculating port numbers

* Fix network cleanup in case of CKS cluster failure

* Externalize retries and inverval for NSX segment deletion

* Fix CKS scaling when external node(s) present in the cluster

* CKS: Fix port numbers displayed against ETCD nodes

* Add node version details to every node of k8s cluster - as we now support manual upgrade

* Add node version details to every node of k8s cluster - as we now support manual upgrade

* update column name

* CKS: Exclude etcd nodes when calculating port numbers

* update param name

* update param

* UI: Fix CKS cluster creation templates listing for non admins

* CKS: Prevent etcd node start port number to coincide with k8s cluster start port numbers

* CKS: Set default kubernetes cluster node version to the kubernetes cluster version on upgrade

* CKS: Set default kubernetes cluster node version to the kubernetes cluster version on upgrade

* consolidate query

* Fix upgrade logic

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix CKS cluster version upgrade

* CKS: Fix etcd port numbers being skipped

* Fix CKS cluster with etcd nodes on VPC

* Move schema and upgrade for 4.20

* Fix logger

* Fix after rebasing

* Add support for using different CNI plugins with CKS

* Add support for using different CNI plugins with CKS

* remove unused import

* Add UI support and list cni config API

* necessary UI changes

* add license

* changes to support external cni

* UI changes

* Fix NPE on restarting VPC with additional public IPs

* fix merge conflict

* add asnumber to create k8s svc layer

* support cni framework to use as-numbers

* update code

* condition to ignore undefined jinja template variables

* CKS: Do not pass AS number when network ID is passed

* Fix deletion of Userdata / CNI Configuration in projects

* CKS: Add CNI configuration details to the response and UI

* Explicit events for registering cni configuration

* Add Delete cni configuration API

* Fix CKS deployment when using VPC tiers with custom ACLs

* Fix DNS list on VR

* CKS: Use Network offering of the network passed during CKS cluster creation to get the AS number

* CKS cluster with guest IP

* Fix: Use control node guest IP as join IP for external nodes addition

* Fix DNS resolver issue

* Improve etcd indexing - start from 1

* CKS: Add external node to a CKS cluster deployed with etcd node(s) successfully

* CKS: Add external node to a CKS cluster deployed with etcd node(s) successfully

* simplify logic

* Tweak setup-kube-system script for baremetal external nodes

* Consider cordoned nodes while getting ready nodes

* Fix CKS cluster scale calculations

* Set token TTL to 0 (no expire) for external etcd

* Fix missing quotes

* Fix build

* Revert PR 9133

* Add calico commands for ens35 interface

* Address review comments: plan CKS cluster deployment based on the node type

* Add qemu-guest-agent dependency for kvm based templates

* Add marvin test for CKS clusters with different offerings per node type

* Remove test tag

* Add marvin test and fix update template for cks and since annotations

* Fix marvin test for adding and removing external nodes

* Fix since version on API params

* Address review comments

* Fix unit test

* Address review comments

* UI: Make CKS public templates visible to non-admins on CKS cluster creation

* Fix linter

* Fix merge error

* Fix positional parameters on the create kubernetes ISO script and make the ETCD version optional

* fix etcd port displayed

* Further improvements to CKS  (#118)

* Multiple nics support on Ubuntu template

* Multiple nics support on Ubuntu template

* supports allocating IP to the nic when VM is added to another network - no delay

* Add option to select DNS or VR IP as resolver on VPC creation

* Add API param and UI to select option

* Add column on vpc and pass the value on the databags for CsDhcp.py to fix accordingly

* Externalize the CKS Configuration, so that end users can tweak the configuration before deploying the cluster

* Add new directory to c8 packaging for CKS config

* Remove k8s configuration from resources and make it configurable

* Revert "Remove k8s configuration from resources and make it configurable"

This reverts commit d5997033ebe4ba559e6478a64578b894f8e7d3db.

* copy conf to mgmt server and consume them from there

* Remove node from cluster

* Add missing /opt/bin directory requrired by external nodes

* Login to a specific Project view

* add indents

* Fix CKS HA clusters

* Fix build

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>

* Add missing headers

* Fix linter

* Address more review comments

* Fix unit test

* Fix scaling case for the same offering

* Revert "Login to a specific Project view"

This reverts commit 95e37563f48573780b07a038a7f48c0bc04e9b64.

* Revert "Fix CKS HA clusters" (#120)

This reverts commit 8dac16aa359faa6500ea1e1ce548169cfd08331a.

* Apply suggestions from code review about user data

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update api/src/main/java/org/apache/cloudstack/api/command/user/userdata/BaseRegisterUserDataCmd.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Refactor column names and schema path

* Fix scaling for non existing previous offering per node type

* Update node offering entry if there was an existing offering but a global service offering has been provided on scale

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-06-19 11:00:42 +05:30
Daniel Augusto Veronezi Salvador
380884ed50
Fix data being replicated on VM's metadata file in the VR (#10987) 2025-06-13 20:13:09 +02:00
Pearl Dsilva
193d6ef52f Merge branch '4.20' of https://github.com/apache/cloudstack 2025-04-18 18:29:44 -04:00
Pearl Dsilva
a95e9d7f77 Merge branch '4.19' of https://github.com/apache/cloudstack into 4.20 2025-04-18 18:29:16 -04:00
Wei Zhou
f2e688af26
VR: add bind-interfaces to /etc/dnsmasq.d/cloud.conf (#10739) 2025-04-18 13:20:13 +02:00
Daan Hoogland
24b7c66251 Merge branch '4.20' 2025-02-24 14:33:12 +01:00
Wei Zhou
789f94b664
VR: fix duplicated lines in .htaccess (#10254) 2025-02-14 08:57:48 -05:00
Daan Hoogland
0dcb8da03a Merge branch '4.20' 2025-02-12 16:54:05 +01:00
Daan Hoogland
4f3e8e8c5a Merge branch '4.19' into 4.20 2025-02-12 15:00:51 +01:00
Vishesh
ae1d7cc860
Fix private gateway acl on static routes (#10262) 2025-02-10 18:50:30 +01:00
dahn
d453c63848
cleanup VM IP after expunge in redundant VPC (#10183)
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
2025-02-09 09:34:20 +01:00
Wei Zhou
90c960eeed
VPC VR: fix ACL between tier and private gateway (#10268) 2025-02-04 16:00:58 +01:00
John Bampton
0f3ea8ceb2
Add pre-commit hook fix-byte-order-marker (#10147)
https://github.com/pre-commit/pre-commit-hooks?tab=readme-ov-file#fix-byte-order-marker
2025-01-03 09:40:43 +01:00
Daan Hoogland
41c27e116c Merge release branch 4.19 to 4.20
* 4.19:
  VR: fix site-2-site VPN if split connections is enabled (#10067)
  UI: fix cannot open 'Edit tags' modal for static routes (#10065)
2024-12-30 13:07:56 +01:00
Wei Zhou
5c01cff304
VR: fix site-2-site VPN if split connections is enabled (#10067) 2024-12-28 12:09:36 +01:00
Daan Hoogland
b7f0aac519 Merge branch '4.19' into 4.20 2024-12-20 14:34:39 +01:00
Wei Zhou
ed1b145a3a
VR: apply iptables rules when add/remove static routes (#10064) 2024-12-20 16:05:22 +05:30
Daan Hoogland
da54234585 Merge branch '4.19' into 4.20.merge 2024-12-03 16:32:15 +01:00
Rodrigo D. Lopez
4189bac8e0
enable to create VPC portfowarding rules with source cidr (#7081)
Co-authored-by: Lopez <rodrigo@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2024-11-28 17:53:07 +01:00
Daan Hoogland
f7b7339bcf Merge branch '4.19' 2024-11-07 11:33:50 +01:00
Rohit Yadav
5b7c86aa45 Merge remote-tracking branch 'origin/4.18' into 4.19
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-11-07 14:51:43 +05:30
Wei Zhou
896b8bfabf
VR: fix wrong check when compare two configuration files (#9822) 2024-11-07 14:48:02 +05:30
Wei Zhou
04c428cb2f
CKS: add ConfigDrive to cloud-init datasource_list in systemvm template (#7650)
* CKS: add ConfigDrive to cloud-init datasource_list in systemvm template

* systemvm template: update debian 11.7.0 iso url

* CKS: get K8S iso by LABEL=CDROM if config drive ISO is attached

* Revert "CKS: add ConfigDrive to cloud-init datasource_list in systemvm template"

This reverts commit b6863a5ce1b9757d7c5bbf3ba9720e2b61410c7d.

* CKS: patch cloud-init in opt/cloud/bin/setup/cksnode.sh

* PR7650: move ConfigDrive before CloudStack in datasource list

* Revert "CKS: patch cloud-init in opt/cloud/bin/setup/cksnode.sh"

This reverts commit 75be03c6aaf58e3939f59c657d84c6495538d3c2.

* CKS: fix ConfigDrive
2024-09-27 09:09:27 -03:00
Wei Zhou
94a47ac778
VR: fix password server exception when no password is found (#9699)
see errors in /var/log/daemon.log below
```
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]: ----------------------------------------
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]: Exception occurred during processing of request from ('192.168.20.8', 51108)
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]: Traceback (most recent call last):
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:   File "/usr/lib/python3.11/socketserver.py", line 691, in process_request_thread
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:     self.finish_request(request, client_address)
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:   File "/usr/lib/python3.11/socketserver.py", line 361, in finish_request
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:     self.RequestHandlerClass(request, client_address, self)
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:   File "/usr/lib/python3.11/socketserver.py", line 755, in __init__
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:     self.handle()
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:   File "/usr/lib/python3.11/http/server.py", line 432, in handle
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:     self.handle_one_request()
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:   File "/usr/lib/python3.11/http/server.py", line 420, in handle_one_request
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:     method()
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:   File "/opt/cloud/bin/passwd_server_ip.py", line 117, in do_GET
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:     self.wfile.write('saved_password')
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:   File "/usr/lib/python3.11/socketserver.py", line 834, in write
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]:     self._sock.sendall(b)
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]: TypeError: a bytes-like object is required, not 'str'
Sep 13 12:36:58 systemvm passwd_server_ip.py[2154]: ----------------------------------------
```
2024-09-19 10:58:57 -03:00
Wei Zhou
679ce1a639
feature: Dynamic and Static Routing (#9470)
This PR contains 3 features

- IPv4 Static Routing (Routed mode) #9346
Design document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=306153967

- AS Numbers Management #9410
Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/BGP+AS+Numbers+Management


- Dynamic routing
Design Document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=315492858

- Document: https://github.com/apache/cloudstack-documentation/pull/419

Rename nsx mode to routing mode

by
```
git grep -l nsx_mode  |xargs sed -i "s/nsx_mode/routing_mode/g"
git grep -l nsxmode  |xargs sed -i "s/nsxmode/routingmode/g"
git grep -l nsxMode  |xargs sed -i "s/nsxMode/routingMode/g"
git grep -l NsxMode  |xargs sed -i "s/NsxMode/RoutingMode/g"
```
- re-organize sql changes
- fix NPE as rules do not have public ip
- fix missing destination cidr in ingress rules
- disable network usage for routed network
- fix DB exception as network_id is -1 during network creation
- apply ingress/egress routing rules
- VR changes to configure nft rules for isolated network
- VR: setup nft rule for control network
- VR: flush all iptables rules
- fix NPE which is because ingress rules do not have public ip associated
- fix dest cidr is missing in nft tables
- add ip4 routing and ip4 routes to list network and list vpc response
- fix ingress rule is missing when vr is restarted
- fix icmp types in nft rules
- add tab to manage routing firewall rules
- fix ingress rules are not applied when VR is restarted
- add default rules in FORWARD chain
- fix create vpc offerings
- fix public ip is not assigned to vpc
- fix network offering is not listed when create vpc tier
- add is_routing to boot args of vpc vr
- remove table ip4_firewall in vpc vr
- release or remove subnet when remove a network
- implemenent fw_vpcrouter_routing
- fix wrong ip familty when flush ipv4 rules
- fix acl rules are not applied due to wrong version (should be 6 which means ip6 rules are removed)
- add default rules for vpc tiers so that tcp connections (e.g. ssh) work
- append policy rules after default rules
- remove /usr/local/cloud/systemvm/ in routers
- throw an exception when allocate subnet with cidrsize
- fix some TODOs
- add new parameters to update API
- return type Ipv4GuestSubnetNetworkMap when get or create subnet
- fix firewall rules are broken
- add domain_id and account_id to db
- add domain/account/project to ipv4 subnet response
- create ipv4 subnet for domain/account/project
- check conflict when update ipv4 subnet
- ui changes
- add parent subnet to response
- add list for ipv4 subnet
- implement some methods
- fix list subnets for guest networks by zoneid
- UI changes
- fix delete ipv4 subnet for network
- fix ipv4 subnet is set to zone guest network cidr if cidrsize is specified
- add zone info to response if parent subnet is null but network is not
- fix gateway/cidr is not set when create network with cidrsize
- fix order of nft rules in the VRs

* Routed v24

- add classes in marvin base.py

* Routed v25

- add test_01_subnet_zone
- fix dedicate to domain/account failure
- list subnets for network by keyword and subnet

* Routed v26: implement subnet auto-allocation

- add utils for split ip ranges into small subnets
- add utils to get start/end ip of a cidr
- implement subnet auto-generation
- add global settings

* Routed 27: add subnet for VPC

- add db column for vpc_id
- add db record for vpc
- remove db record when delete a vpc
- add checkConflicts methods
- remove duplicated settings
- check ipv4 cidr when create subnet

* Routed v28: update smoke tests

- update test_ipv4_routing.py
- search subnets by networkid

* Routed 29: fix vpc and add more tests

- fix createnetwork in vpc
- add vpc id/name to response
- fix zone id/name are not displayed in some cases
- add smoke test for vpc
- add smoke tests for failed cases
- add smoke test for connectivity checks
- marvin: add "-q" to ssh command

* Routed 31: ui and smoke tests

- UI: add link to network in list view
- add nftables rules check in VRs

* Routed 32: add chain OUTPUT and more rules

- fix the issue 80/443/8080 is not reachable from VR itself

```
2024-06-27 10:21:52,121 INFO     Executing: systemctl start cloud-password-server@172.31.1.1
2024-06-27 10:21:52,128 INFO     Service cloud-password-server@172.31.1.1 start
2024-06-27 10:21:52,129 INFO     Executing: ps aux
2024-06-27 10:24:02,175 ERROR    Failed to update password server due to: <urlopen error [Errno 110] Connection timed out>
```

* Routed: fix dns search from VMs in Isolated networks

* Routed: fix VPC dns issue due to gateway IP is missing in cloud.conf

This is caused by NSX integration, and fixed by
https://github.com/apache/cloudstack/pull/9102/

* Routed: rename routing_mode to network_mode

* Routed: replace centos5.5 template in smoke test as dhclient does not work in the vms

// this does not work
refer to https://dominikrys.com/posts/disable-udp-checksum-validation/#ignoring-udp-checksums-with-nftables
and
https://forum.openwrt.org/t/udp-checksum-with-nftables/161522/11

the vm should have checksum offloading disabled

* Routed: fix smoke test due to wrong cidrlist of egress rules and missing ingress rule from VR

* PR 9346: fix lint error schema-41910to42000.sql

* PR 9346: ui polish v1

* PR 9346: create VPC with cidrsize

* Routed: fix test failures with test_network_ipv6 and test_vpc_ipv6 due to 'ssh -q'

* Routed: fix /usr/local/cloud/systemvm/ are removed after SSVM/CPVM reboot

* Routed: fix IP of additional nics of VPC VR is not gateway

* PR 9346: fix cidrsize check when create VPC with cidrsize

* Routed: fix test/integration/smoke/test_ipv4_routing.py:279:16: E713 test for membership should be 'not in'

* PR9346: fix/Update api

* PR 9346: set response object name

* PR9346: UI refactor and small fixes

* PR9346: change return type of getNetworkMode

* PR9346: move IPv4 subnet to seperated tab

* PR9346: revert IpRangesTabGuest.vue back to original

* PR9346: fix remove ipv4 subnet on UI

* PR9346: fix test_ipv4_routing.py

* AS Number Range Management

* Create AS Number Range for a Zone

* Fix build

* Add ListASNRange and fix create ASN range

* Add List AS numbers

* Add UI for AS Numbers

* Fix UI and filter AS Numbers

* Add AS Number on Isolated network creation and refactor UI and response

* Release AS Number

* Add network offering new columns

* Add UI support to view and add AS number and configure network offering

* Automatically assign AS Number if not specify AS number

* update variable name

* Fix routing mode check

* UI: Only allow selecting AS number when routing mode is Dynamic and specifyAsNumber is true

* UI: Only pass AS number when supported by the network offering

* Release AS number on network deletion

* Add deleteASNRange command (#81)

* API: List ASNumbers by asnumber (#83)

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* AS number management extensions

* Support AS number on VPC tier creation based on the offering

* Fix delete AS Range

* Fix UI values

* UI: Minor fix for releasing AS number

* UI: Move management of AS Range to Zone details view

* Fix specify_as_number column in network_offering table to set the default false

* Add events for AS number operations

* Allow users to list AS Numbers and fix network form for Normal users

* Add AS number details to list networks response

* Fix Allocated time format

* Fix Allocated time format

* support in details view too

* Fix: Do not release AS number if acquired network requires AS number

* Fix: Do not release AS number if acquired network requires AS number

* Fix typo

* Fix allocated release

* Fix event type

* UI: Add Routing mode and Specify AS to the network offering details

* UI: Add Routing mode and Specify AS to the network offering details

* Address comment

* Fix release AS number of network deletion

* Fix release AS number of network deletion

* Fix

* Restore release to its place based on the boolean

* Rename boolean

* API: Add networkId as listASNumber parameter

* Add Network name to the search view filter for AS numbers

* Present allocated time in human readable format - Pubilc IP / AS Numbers

* Add account / domain filter for AS numbers

* Add support for AS numbers on VPC offerings

* Refactor AS number allocation to VPC and non VPC isolated networks

* Checkstyle

* Add support for AS numbers on VPC offerings

* extend vpc offering view and vpcoffering response

* merge https://github.com/shapeblue/cloudstack-playtika/pull/115 and change network_id of as_numbers to include vpc_id

* Display AS number of VPC tiers as the AS number of the VPC

* extend asnumber response and ui support

* improve UI and as number response to view VPC details

* List only dynamic offerings for vpc tiers with specify as numbers

* Fix release AS number

* Fix AS number displayed as 0 when no AS number assigned

* Fix VPC offering creation without specify AS

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix release AS number on VPC deletion

* Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java

* Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java

* Fix missing column on asnumber table

* Fix listASNumbers API to support vpcid and obtain AS number from vpc for tiers

* Prevent listing 0 AS number for VPC

* Fix create Isolated Network form

* Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

* Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

* Dynamic: move routingmode/specifyasn after networkmode in AddNetworkOffering.vue on UI

* Dynamic: fix ip4routing in network response

* Dynamic/systemvm: add FRR to systemvm template

* Dynamic: BGP peers (DB,VO,Dao)

* Dynamic: BGP peers (VR/server)

* Dynamic: v3

- remove BgpPeer class
- fix vpc vr has bgp peers of only 1 tier
- rename ip4_cidr to guest_ip4_cidr
- rename ip6_cidr to guest_ip6_cidr
- generate /etc/frr/frr.conf
- apply BGP peers on Dynamic-Routed network even if there is no BGP peers

* Dynamic v4: fix vpc vr

- fix duplicated guest cidr in frr.conf in vpc vr

todo
- restart frr / reload frr (reload will cause bgp session to Policy state)
- apis for bgp peers
- assign/release bgp peer from/to network

* Dynamic v5: add apis for bgp peers

* Dynamic v6: fix bugs

- set response object name
- remove required as number when update
- fix checks when update
- allow regular users to list bgp peers

* Dynamic v7: move apis to bgp sub-dir

* Dynamic v8: add tab for manage BGP peers on UI

* Dynamic v9: fix update bgp with same config

* Dynamiv v10: add changeBgpPeersForNetworkCmd

* Dynamic v11: create network with bgppeerids

- create network with bgppeerids
- add marvin classes
- add smoke tests
- remove uuid from bgp_peer_network_map
- fix created/removed in bgp_peer_network_map
- remove bgppeers when remove a network
- UI: fix delete bgp peer

* Dynamic v12: add test for vpc tiers

* Dynamic v13: bug fixes

- fix change BGP peers for network in Allocated state
- fix listing network returns removed record
- fix all vpc tiers have the same settings
- remove BGP peers as part of network removal
- remove FRR settings for vpc tiers without any BGP peers
- UI: fix no error msg when change BGP peers

* Dynamic v14: assign BGP Peers for VPC instead of VPC tiers

- create vpc with bgppeerids
- do not allow create/update vpc tier with bgppeerids
- apply all bgp peers when create/delete a vpc tier
- UI: change bgp peers for vpc
- test: update tests on vpc

* Dynamic: fix build errors after merging as number PR

* Dynamic: fix TODOs

* Dynamic: fix smoke test on VPC

* Allow creation of networks by users with as numbers

* Address review comments

* Move BGPService to bgp package and inject it on BaseCmd

* Revert changes for CKS and address more comments

* Display left side menu option for AS number only for root admin

* Dynamic: create/update BGP peer with details

refer to https://docs.frrouting.org/en/latest/bgp.html

* Dynamic: fix build error and remove access to ListBgpPeers cmd for regular users

* Dynamic: assign all zone BGP peers to user networks

* Dynamic: show BGP peer info of networks only for root admin

* AS number: disable specifyasnumber for non-NSX offerings

* Dynamic: pass bgppeer details to command and fix typo with ip6 addr

* Dynamic: list BGP peers by isdedicated, and fix change bgppeers for network/vpc

* Dynamic: add UI labels

* Dynamic: add bgp peers to vpc response

* Dynamic: list bgp peers by keyword, fix list by asnumber

* Dynamic: fix list bgppeers by keyword and db schema

* Dynamic: fix list bgppeers do not return dedicated peers

* Dynamic: update UI when create network/vpc offering

* Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* Update tools/marvin/setup.py

* Dynamic: network mode must be same when update a network with new offering

* Dynamic: add method networkModel.isAnyServiceSupportedInNetwork

* Dynamic: rename APIs and classes

* Dynamic: fix unit tests due to previous changes

* Dynamic: validateNetworkCidrSize when auto-create subnet

* Dynamic: check AS number overlap

* Dynamic: add ActionEvent

* Dynamic: small code optimization

* Dynamic: fix ui bugs after api rename

* Dynamic: add marvin and test for ASN ranges and AS numbers

* Dynamic: add account setting use.system.bgp.peers

also
- change the default value of routed.ipv4.vpc.max.cidr.size and routed.ipv4.vpc.min.cidr.size
- change the category of settings

* static: fix ui error when delete zone ipv4 subnets

* static: small UI polish

* Dynamic: throw exception when as number is required but not passed

* Dynamic: fix typo when create FRR directory which causes network deletion failures

* Dynamic: connect to ALL (or ALL dedicated) BGP peers if no BGP peer mapping for the network/vpc

* Dynamic: throw exception when as number is required for VPC but not passed

* Dynamic: list bgp peers by useSystemBgpPeers

* Dynamic: fix frr config in VPC VR when change bgp peers

* Dynamic: create frr config even if there is no VPC tiers

* Dynamic: list bgp peers by zoneid (required for account) and account

* Dynamic: only apply FRR config for vpc tiers with dynamic routing

* Dynamic: donot send commands to router if commands size is 0

* Dynamic: fix 'new IPv6 address is not valid' when update bgp peer without IPv6

* Dynamic: throw exception if fail to allocate AS number when create network/vpc with dynamic routing

* Dynamic: enable ipv6 unicast and 'ip nht resolve-via-default'

* Dynamic: delete network/vpc if fail to allocate AS number when create network/vpc with dynamic routing

* test: add unit tests for ASN APIs

* test: add unit tests for core module

* test: add unit tests for API responses

* test: add unit tests for BgpPeerTO

* test: add minor changes

* test: add tests for create/delete/update/list RoutingFirewallRuleCmd

* Static: show ip4 routes for vpc tiers

* test: fix smoke test failure caused by type change of as number

* test: add test for Ipv4SubnetForZoneCmd

* test: add test for Ipv4SubnetForGuestNetworkCmd and BgpPeerCmd

* UI: do not show redundant router when network mode is ROUTED as RVR is not supported

* UI: hide 'Conserve mode' when networkmode is ROUTED

* test: add unit tests for ListASNumbersCmdTest

* Static: remove allocated IPv4 subnet when delete a network or vpc

* test: add unit tests for BgpPeersRules

* Dynamic: set ipv4routing from network offering

* server: list as numbers and ipv4 subnets by keyword

* server: remove dedicated bgp peers and ipv4 subnets when delete an account or domain

* server: fix dedicated ipv4 subnet is allocated to other accounts

* UI: fix allocated time format

* server: ignore project is projectid is -1 so bgppeers/ipv4subnets works in project view

* UI: add project column to bgp peers and ipv4 subnets

* server: fix list AS numbers by domain admin or normal user

* server: fix network creation when ipv4 subnet is dedicated

* UI: polish network.js

* Dynamic: fix frr config for ipv6 routing

* Static routing: support cks cluster

* Static: get/create IPv4 subnet from dedicated subnets at first

* Dynamic: add BGP peers tab

* Static: remove redundant loops

* api: add since to api and response

* server: add unit tests

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-06 08:55:17 +05:30
Abhisar Sinha
605534b417
feature: Shared Storage Filesystem as a First Class Feature (#9208)
This PR implements Storage filesystem as a first class feature.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Filesystem+as+a+First+Class+Feature

Documentation PR: apache/cloudstack-documentation#420

Co-authored-by: Wei Zhou <weizhou@apache.org>
2024-09-05 17:22:32 +05:30
Rohit Yadav
60c9c9adb6 Merge remote-tracking branch 'origin/4.19'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-04 13:20:42 +05:30
Wei Zhou
1ca9a10912
VR: remove vpn user info when apply vpn users list (#9568)
Prior to this PR
```
root@r-663-VM:/var/cache/cloud# gzip -dk vpn_user_list.json.aae73e2c-32ba-44f3-bf47-426933a67bcb.gz
root@r-663-VM:/var/cache/cloud# /opt/cloud/bin/update_config.py vpn_user_list.json.aae73e2c-32ba-44f3-bf47-426933a67bcb
{'id': 'vpnuserlist', 'test': {'add': True, 'password': 'test', 'user': 'test'}}
{'vpn_users': [{'user': 'test', 'password': 'test', 'add': True}], 'type': 'vpnuserlist', 'delete_from_processed_cache': False}
 line = # Secrets for authentication using CHAP

 line = # client	server	secret			IP addresses

 line =

 line =

 line = test * test *
```

with this PR
```
root@r-663-VM:/var/cache/cloud# gzip -dk vpn_user_list.json.aae73e2c-32ba-44f3-bf47-426933a67bcb.gz
root@r-663-VM:/var/cache/cloud# /opt/cloud/bin/update_config.py vpn_user_list.json.aae73e2c-32ba-44f3-bf47-426933a67bcb
root@r-663-VM:/var/cache/cloud#
```
2024-09-04 11:57:28 +05:30
Suresh Kumar Anaparti
f9c4edc66f
SystemVM template changes - updated debian version & other changes (#9625)
* systemvm template changes, to explicitly update the module dependencies, vmware hardware version updated to 13, fsck added in grub

* Update systemvm template debian version to 12.7
2024-09-04 11:42:41 +05:30
Wei Zhou
1662af131e
[4.20] VR: fix issue if userdata is binary data (#8739) 2024-07-25 10:04:58 +02:00
John Bampton
806692bcea
pre-commit: enable Python flake8 rule E272 (#9361)
There should be only one space before a keyword.
2024-07-10 09:17:25 +02:00
John Bampton
c6d3f6e623
pre-commit: enable Python flake8 rule E271 (#9358) 2024-07-09 10:17:53 +02:00
Daan Hoogland
373f017002 Merge branch '4.19' 2024-06-18 19:58:43 +02:00
Daan Hoogland
050ee44137 Merge branch '4.18' into 4.19 2024-06-18 16:05:45 +02:00
Vishesh
6c7426e3a7
Prevent network disruption on adding a VPC tier for redundant VRs (#9251) 2024-06-18 13:47:45 +02:00
dahn
56b69b157e
prevent duplicate ip table rules in SSVM (#8530)
Co-authored-by: Wei Zhou <weizhou@apache.org>
2024-06-14 09:01:15 +02:00