apache/cloudstack
1
0
Fork 0
mirror of https://github.com/apache/cloudstack.git synced 2025-10-26 08:42:29 +01:00
Code Issues Packages Projects Releases Wiki Activity

Upgrade to JRE17 and Upgrade System VMs/VRs to Python3 and Debian 12 (#8497)

Browse Source 
* Update to 4.20.0

* Update to python3

* Upgrade to JRE 17

* Upgrade to Debian 12.4.0

* VR: upgrade to python3

for f in `find systemvm/ -name *.py`;do
    if grep "print " $f >/dev/null;then
        2to3-2.7 -w $f
    else
        2to3-2.7 -p -w $f
    fi
done

* java: Use JRE17 in cloudstack packages and systemvmtemplate

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* Add --add-opens to JAVA_OPTS in systemd config

* Add --add-opens to JAVA_OPTS in systemd config for usage

* python3: fix "TypeError: a bytes-like object is required, not 'str'"

* python3: fix "ValueError: must have exactly one of create/read/write/append mode"

* Add --add-exports=java.base/sun.security.x509=ALL-UNNAMED for management server

* Use pip3 instead of pip for centos8

* python3: fix "TypeError: write() argument must be str, not bytes"

```
root@r-1037-VM:~# /opt/cloud/bin/passwd_server_ip.py 10.1.1.1
Traceback (most recent call last):
  File "/opt/cloud/bin/passwd_server_ip.py", line 201, in <module>
    serve()
  File "/opt/cloud/bin/passwd_server_ip.py", line 187, in serve
    initToken()
  File "/opt/cloud/bin/passwd_server_ip.py", line 60, in initToken
    f.write(secureToken)
TypeError: write() argument must be str, not bytes
root@r-1037-VM:~#
```

* Python3: fix "name 'file' is not defined"

```
root@r-1037-VM:~# /opt/cloud/bin/passwd_server_ip.py 10.1.1.1
Traceback (most recent call last):
  File "/opt/cloud/bin/passwd_server_ip.py", line 201, in <module>
    serve()
  File "/opt/cloud/bin/passwd_server_ip.py", line 188, in serve
    loadPasswordFile()
  File "/opt/cloud/bin/passwd_server_ip.py", line 67, in loadPasswordFile
    with file(getPasswordFile()) as f:
NameError: name 'file' is not defined
```

* python3: fix "TypeError: write() argument must be str, not bytes" (two more files)

* Upgrade jaxb version

* python3: fix more "TypeError: a bytes-like object is required, not str"

* python3: fix "Failed to update password server"

Failed to update password server due to: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

* python3: fix "bad duration value: ikelifetime=24.0h"

Jan 15 13:57:20 systemvm ipsec[3080]: # bad duration value: ikelifetime=24.0h

* python3: fix password server "invalid save_password token"

* test: incease retries in test_vpc_vpn.py

* python3: fix passwd_server_ip.py

see error below
```
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: ----------------------------------------
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: Exception occurred during processing of request from ('10.1.1.129', 32782)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: Traceback (most recent call last):
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:   File "/usr/lib/python3.9/socketserver.py", line 650, in process_request_thread
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:     self.finish_request(request, client_address)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:   File "/usr/lib/python3.9/socketserver.py", line 360, in finish_request
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:     self.RequestHandlerClass(request, client_address, self)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:   File "/usr/lib/python3.9/socketserver.py", line 720, in __init__
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:     self.handle()
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:   File "/usr/lib/python3.9/http/server.py", line 427, in handle
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:     self.handle_one_request()
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:   File "/usr/lib/python3.9/http/server.py", line 415, in handle_one_request
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:     method()
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:   File "/opt/cloud/bin/passwd_server_ip.py", line 120, in do_GET
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:     self.wfile.write(password)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:   File "/usr/lib/python3.9/socketserver.py", line 799, in write
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]:     self._sock.sendall(b)
Jan 15 18:51:21 systemvm passwd_server_ip.py[1507]: TypeError: a bytes-like object is required, not 'str'
```

* python3: fix self.cl.get_router_password in Redundant VRs

```
File "/opt/cloud/bin/cs/CsDatabag.py", line 154, in get_router_password
    md5.update(passwd)
TypeError: Unicode-objects must be encoded before hashing"]
```

* scripts: mark multipath scripts as executable

* systemvm template: remove hyperv packages and do not export

* VR: update default RAM size of System VMs/VRs to 512MiB

Before
```
mysql> select id,name,cpu,speed,ram_size,unique_name,system_use from service_offering where name like "System%";
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
| id | name                                                     | cpu  | speed | ram_size | unique_name                      | system_use |
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
|  3 | System Offering For Software Router                      |    1 |   500 |      256 | Cloud.Com-SoftwareRouter         |          1 |
|  4 | System Offering For Software Router - Local Storage      |    1 |   500 |      256 | Cloud.Com-SoftwareRouter-Local   |          1 |
|  5 | System Offering For Internal LB VM                       |    1 |   256 |      256 | Cloud.Com-InternalLBVm           |          1 |
|  6 | System Offering For Internal LB VM - Local Storage       |    1 |   256 |      256 | Cloud.Com-InternalLBVm-Local     |          1 |
|  7 | System Offering For Console Proxy                        |    1 |   500 |     1024 | Cloud.com-ConsoleProxy           |          1 |
|  8 | System Offering For Console Proxy - Local Storage        |    1 |   500 |     1024 | Cloud.com-ConsoleProxy-Local     |          1 |
|  9 | System Offering For Secondary Storage VM                 |    1 |   500 |      512 | Cloud.com-SecondaryStorage       |          1 |
| 10 | System Offering For Secondary Storage VM - Local Storage |    1 |   500 |      512 | Cloud.com-SecondaryStorage-Local |          1 |
| 11 | System Offering For Elastic LB VM                        |    1 |   128 |      128 | Cloud.Com-ElasticLBVm            |          1 |
| 12 | System Offering For Elastic LB VM - Local Storage        |    1 |   128 |      128 | Cloud.Com-ElasticLBVm-Local      |          1 |
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
10 rows in set (0.00 sec)
```

New value
```
mysql> select id,name,cpu,speed,ram_size,unique_name,system_use from service_offering where name like "System%";
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
| id | name                                                     | cpu  | speed | ram_size | unique_name                      | system_use |
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
|  3 | System Offering For Software Router                      |    1 |   500 |      512 | Cloud.Com-SoftwareRouter         |          1 |
|  4 | System Offering For Software Router - Local Storage      |    1 |   500 |      512 | Cloud.Com-SoftwareRouter-Local   |          1 |
|  5 | System Offering For Internal LB VM                       |    1 |   256 |      512 | Cloud.Com-InternalLBVm           |          1 |
|  6 | System Offering For Internal LB VM - Local Storage       |    1 |   256 |      512 | Cloud.Com-InternalLBVm-Local     |          1 |
|  7 | System Offering For Console Proxy                        |    1 |   500 |     1024 | Cloud.com-ConsoleProxy           |          1 |
|  8 | System Offering For Console Proxy - Local Storage        |    1 |   500 |     1024 | Cloud.com-ConsoleProxy-Local     |          1 |
|  9 | System Offering For Secondary Storage VM                 |    1 |   500 |      512 | Cloud.com-SecondaryStorage       |          1 |
| 10 | System Offering For Secondary Storage VM - Local Storage |    1 |   500 |      512 | Cloud.com-SecondaryStorage-Local |          1 |
| 11 | System Offering For Elastic LB VM                        |    1 |   128 |      512 | Cloud.Com-ElasticLBVm            |          1 |
| 12 | System Offering For Elastic LB VM - Local Storage        |    1 |   128 |      512 | Cloud.Com-ElasticLBVm-Local      |          1 |
+----+----------------------------------------------------------+------+-------+----------+----------------------------------+------------+
10 rows in set (0.01 sec)
```

* debian12: fix test_network_ipv6 and test_vpc_ipv6

* python3: remove duplicated imports

* debian12: failed to start Apache2 server (SSLCipherSuite @SECLEVEL=0)

error message
```
[Sat Jan 20 22:51:14.595143 2024] [ssl:emerg] [pid 10200:tid 140417063888768] AH02562: Failed to configure certificate cloudinternal.com:443:0 (with chain), check /etc/ssl/certs/cert_apache.crt
[Sat Jan 20 22:51:14.595234 2024] [ssl:emerg] [pid 10200:tid 140417063888768] SSL Library Error: error:0A00018E:SSL routines::ca md too weak
AH00016: Configuration Failed
```

openssl version
```
root@s-167-VM:~# openssl version -a
OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)
built on: Mon Oct 23 17:52:22 2023 UTC
platform: debian-amd64
options:  bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-3.0.11=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-3"
MODULESDIR: "/usr/lib/x86_64-linux-gnu/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0x80202001478bfffd:0x0
```

certificate
```
root@s-167-VM:~# keytool -printcert -rfc -file /usr/local/cloud/systemvm/certs/realhostip.crt
-----BEGIN CERTIFICATE-----
MIIFZTCCBE2gAwIBAgIHKBCduBUoKDANBgkqhkiG9w0BAQUFADCByjELMAkGA1UE
BhMCVVMxEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxGjAY
BgNVBAoTEUdvRGFkZHkuY29tLCBJbmMuMTMwMQYDVQQLEypodHRwOi8vY2VydGlm
aWNhdGVzLmdvZGFkZHkuY29tL3JlcG9zaXRvcnkxMDAuBgNVBAMTJ0dvIERhZGR5
IFNlY3VyZSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTERMA8GA1UEBRMIMDc5Njky
ODcwHhcNMTIwMjAzMDMzMDQwWhcNMTcwMjA3MDUxMTIzWjBZMRkwFwYDVQQKDBAq
LnJlYWxob3N0aXAuY29tMSEwHwYDVQQLDBhEb21haW4gQ29udHJvbCBWYWxpZGF0
ZWQxGTAXBgNVBAMMECoucmVhbGhvc3RpcC5jb20wggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQCDT9AtEfs+s/I8QXp6rrCw0iNJ0+GgsybNHheU+JpL39LM
TZykCrZhZnyDvwdxCoOfE38Sa32baHKNds+y2SHnMNsOkw8OcNucHEBX1FIpOBGp
h9D6xC+umx9od6xMWETUv7j6h2u+WC3OhBM8fHCBqIiAol31/IkcqDxxsHlQ8S/o
CfTlXJUY6Yn628OA1XijKdRnadV0hZ829cv/PZKljjwQUTyrd0KHQeksBH+YAYSo
2JUl8ekNLsOi8/cPtfojnltzRI1GXi0ZONs8VnDzJ0a2gqZY+uxlz+CGbLnGnlN4
j9cBpE+MfUE+35Dq121sTpsSgF85Mz+pVhn2S633AgMBAAGjggG+MIIBujAPBgNV
HRMBAf8EBTADAQEAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAOBgNV
HQ8BAf8EBAMCBaAwMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5nb2RhZGR5
LmNvbS9nZHMxLTY0LmNybDBTBgNVHSAETDBKMEgGC2CGSAGG/W0BBxcBMDkwNwYI
KwYBBQUHAgEWK2h0dHA6Ly9jZXJ0aWZpY2F0ZXMuZ29kYWRkeS5jb20vcmVwb3Np
dG9yeS8wgYAGCCsGAQUFBwEBBHQwcjAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Au
Z29kYWRkeS5jb20vMEoGCCsGAQUFBzAChj5odHRwOi8vY2VydGlmaWNhdGVzLmdv
ZGFkZHkuY29tL3JlcG9zaXRvcnkvZ2RfaW50ZXJtZWRpYXRlLmNydDAfBgNVHSME
GDAWgBT9rGEyk2xF1uLuhV+auud2mWjM5zArBgNVHREEJDAighAqLnJlYWxob3N0
aXAuY29tgg5yZWFsaG9zdGlwLmNvbTAdBgNVHQ4EFgQUZyJz9/QLy5TWIIscTXID
E8Xk47YwDQYJKoZIhvcNAQEFBQADggEBAKiUV3KK16mP0NpS92fmQkCLqm+qUWyN
BfBVgf9/M5pcT8EiTZlS5nAtzAE/eRpBeR3ubLlaAogj4rdH7YYVJcDDLLoB2qM3
qeCHu8LFoblkb93UuFDWqRaVPmMlJRnhsRkL1oa2gM2hwQTkBDkP7w5FG1BELCgl
gZI2ij2yxjge6pOEwSyZCzzbCcg9pN+dNrYyGEtB4k+BBnPA3N4r14CWbk+uxjrQ
6j2Ip+b7wOc5IuMEMl8xwTyjuX3lsLbAZyFI9RCyofwA9NqIZ1GeB6Zd196rubQp
93cmBqGGjZUs3wMrGlm7xdjlX6GQ9UvmvkMub9+lL99A5W50QgCmFeI=
-----END CERTIFICATE-----

Warning:
The certificate uses the SHA1withRSA signature algorithm which is considered a security risk. This algorithm will be disabled in a future update.
```

it comes from
```
$ openssl x509 -in ./systemvm/agent/certs/realhostip.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 11277268652730408 (0x28109db8152828)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certificates.godaddy.com/repository, CN = Go Daddy Secure Certification Authority, serialNumber = 07969287
        Validity
            Not Before: Feb  3 03:30:40 2012 GMT
            Not After : Feb  7 05:11:23 2017 GMT
        Subject: O = *.realhostip.com, OU = Domain Control Validated, CN = *.realhostip.com
```

* debian12: use ed25519 instead of rsa as ssh-rsa has been deprecated in OpenSSH

on xenserver
```
[root@pr8497-t8906-xenserver-71-xs2 ~]# ssh -i .ssh/id_rsa.cloud -p 3922 169.254.214.153
Warning: Permanently added '[169.254.214.153]:3922' (ECDSA) to the list of known hosts.
Permission denied (publickey).
```
in the CPVM
Jan 22 19:31:09 v-1-VM sshd[2869]: userauth_pubkey: signature algorithm ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Jan 22 19:31:09 v-1-VM sshd[2869]: Connection closed by authenticating user root 169.254.0.1 port 54704 [preauth]
```

ssh-dss (DSA) is not supported either

* debian12: add PubkeyAcceptedAlgorithms=+ssh-rsa to sshd_config

* VR: install python3 packages in case of Debian 11

* pom.xml: exclude systemvm/agent/packages/* in license check

* systemvm: do not patch router/systemvm during startup

this will cause 4.19 SYSTEM template not work, but may be expected
- python3 VS python2 (default)
- openSSL 3.0.1 VS 1.1.1w
- openssh-server 9.1 VS 8.4

* VR: patch router/systemvm if template is debian11

This supports debian 11 template by
- revert change in systemvm/debian/etc/ssh/sshd_config
- patch VR/systemvms during startup
- install packages during patching system vm/routers

* python3 flake: fix E502 the backslash is redundant between brackets

```
../debian/root/health_checks/router_version_check.py:55:70: E502 the backslash is redundant between brackets
../debian/root/health_checks/router_version_check.py:58:61: E502 the backslash is redundant between brackets
../debian/root/health_checks/router_version_check.py:67:71: E502 the backslash is redundant between brackets
../debian/root/health_checks/router_version_check.py:70:60: E502 the backslash is redundant between brackets
../debian/root/health_checks/haproxy_check.py:47:71: E502 the backslash is redundant between brackets
../debian/root/health_checks/haproxy_check.py:48:64: E502 the backslash is redundant between brackets
../debian/root/health_checks/cpu_usage_check.py:43:54: E502 the backslash is redundant between brackets
../debian/root/health_checks/cpu_usage_check.py:46:58: E502 the backslash is redundant between brackets
../debian/root/health_checks/memory_usage_check.py:31:65: E502 the backslash is redundant between brackets
../debian/root/health_checks/memory_usage_check.py:42:57: E502 the backslash is redundant between brackets
../debian/root/health_checks/memory_usage_check.py:45:63: E502 the backslash is redundant between brackets
```

* python3 flake: fix E275 missing whitespace after keyword

```
../debian/opt/cloud/bin/cs_firewallrules.py:29:20: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_dhcp.py:27:16: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_dhcp.py:36:16: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_guestnetwork.py:33:20: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_guestnetwork.py:35:16: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_vpnusers.py:37:16: E275 missing whitespace after keyword
../debian/opt/cloud/bin/merge.py:230:11: E275 missing whitespace after keyword
../debian/opt/cloud/bin/merge.py:239:19: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_remoteaccessvpn.py:24:12: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs_site2sitevpn.py:24:12: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs/CsHelper.py:90:15: E275 missing whitespace after keyword
../debian/opt/cloud/bin/cs/CsAddress.py:367:15: E275 missing whitespace after keyword
```

* python3 flake: fix configure.py

```
../debian/opt/cloud/bin/configure.py:24:22: E401 multiple imports on one line
../debian/opt/cloud/bin/configure.py:43:180: E501 line too long (294 > 179 characters)
../debian/opt/cloud/bin/configure.py:46:1: E302 expected 2 blank lines, found 1
../debian/opt/cloud/bin/configure.py:63:1: E302 expected 2 blank lines, found 1
../debian/opt/cloud/bin/configure.py:65:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/configure.py:72:1: E302 expected 2 blank lines, found 1
../debian/opt/cloud/bin/configure.py:310:25: E711 comparison to None should be 'if cond is not None:'
../debian/opt/cloud/bin/configure.py:312:29: E711 comparison to None should be 'if cond is None:'
../debian/opt/cloud/bin/configure.py:378:25: E711 comparison to None should be 'if cond is not None:'
../debian/opt/cloud/bin/configure.py:380:29: E711 comparison to None should be 'if cond is None:'
../debian/opt/cloud/bin/configure.py:490:29: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
../debian/opt/cloud/bin/configure.py:642:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/configure.py:644:18: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/configure.py:1416:1: E305 expected 2 blank lines after class or function definition, found 1
```

* python3 flake: fix other python files

```
../debian/opt/cloud/bin/vmdata.py:97:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
../debian/opt/cloud/bin/vmdata.py:99:14: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`

../debian/opt/cloud/bin/cs/CsRedundant.py:438:53: E203 whitespace before ':'
../debian/opt/cloud/bin/cs/CsRedundant.py:461:53: E203 whitespace before ':'
../debian/opt/cloud/bin/cs/CsRedundant.py:499:5: E303 too many blank lines (2)

../debian/opt/cloud/bin/cs/CsDatabag.py:189:1: E302 expected 2 blank lines, found 1
../debian/opt/cloud/bin/cs/CsDatabag.py:193:37: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`

../debian/opt/cloud/bin/cs/CsHelper.py:118:30: E231 missing whitespace after ','
../debian/opt/cloud/bin/cs/CsHelper.py:119:15: E225 missing whitespace around operator
../debian/opt/cloud/bin/cs/CsHelper.py:127:19: E225 missing whitespace around operator

../debian/opt/cloud/bin/cs/CsAddress.py:324:43: E221 multiple spaces before operator

../debian/opt/cloud/bin/cs/CsVpcGuestNetwork.py:28:1: E302 expected 2 blank lines, found 1
```

* python3 flake: fix CsNetfilter.py

```
../debian/opt/cloud/bin/cs/CsNetfilter.py:226:13: E117 over-indented
../debian/opt/cloud/bin/cs/CsNetfilter.py:233:180: E501 line too long (197 > 179 characters)
../debian/opt/cloud/bin/cs/CsNetfilter.py:241:14: E201 whitespace after '{'
../debian/opt/cloud/bin/cs/CsNetfilter.py:242:14: E201 whitespace after '{'
../debian/opt/cloud/bin/cs/CsNetfilter.py:247:18: E201 whitespace after '{'
../debian/opt/cloud/bin/cs/CsNetfilter.py:247:74: E202 whitespace before '}'
../debian/opt/cloud/bin/cs/CsNetfilter.py:248:18: E201 whitespace after '{'
```

* systemvm/test: fix sys.path

```
$ bash runtests.sh
/usr/bin/python
Python 3.10.12
Running pycodestyle to check systemvm/python code for errors
Running pylint to check systemvm/python code for errors
Python 3.10.12
pylint 2.12.2
astroid 2.9.3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

Running systemvm/python unit tests
....Device "eth0" does not exist.
.....................
----------------------------------------------------------------------
Ran 25 tests in 0.008s

OK
```

* Revert "systemvm template: remove hyperv packages and do not export"

This reverts commit 4383d59d031bde6eae7ebba261ff641ca0a66cd5.

* debian12: move SQL change to schema-41900to42000.sql

* debian12: update systemvm template version to 4.20 in pom.xml

* pom.xml: fix NPE if templates do not exist on download.cloudstack.org

* debian12: increase default system offering for routers to 384MiB RAM

* CKS: fix addkubernetessupportedversion failed with JRE17

```
marvin.cloudstackException.CloudstackAPIException: Execute cmd: addkubernetessupportedversion failed, due to: errorCode: 530, errorText:Cannot invoke "org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine$State.toString()" because the return value of "com.cloud.api.query.vo.TemplateJoinVO.getState()" is null
```

* python3: revert changes by 2to3 with systemvm/debian/root/health_checks/*.py

* debian12: use ISO/packages on download.cloudstack.org

* VR: Update default ram size to 384

* debian12: fix router_version_check.py after VR live-patch and add health check in test_routers.py

* debian12: fix build error after log4j 2.x merge

* VR: Update default ram size to 512MB (again)

This reverts commit 578dd2b73f380e8231ae1eb59827230757cac5e8 and efafa8c4d63775653a2cd406fca10784fbcec3e3.

* systemvmtemplate: Upgrade to Debian 12.5.0

* systemvm template: increase swap to 512MB

* VR: fix health check error due to deprecated SafeConfigParser

warning below
```
root@r-20-VM:~# /opt/cloud/bin/getRouterMonitorResults.sh true
/root/monitorServices.py:59: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in Python 3.12. Use ConfigParser directly instead.
  parser = SafeConfigParser()
```

* test: fix wget does not work in macchinina vms on vmware80u1

fixes error below
```
{Cmd: wget -t 1 -T 1 www.google.com via Host: 10.0.55.186} {returns: ["wget: '/usr/lib/libpcre.so.1' is not an ELF file", "wget: can't load library 'libpcre.so.1'"]}
```

* packaging: add message for VR memory upgrade after packages installation

---------

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Vishesh <vishesh92@gmail.com>
This commit is contained in:
Wei Zhou 2024-02-26 13:37:50 +01:00 committed by GitHub
parent 0926e5c15a
commit 87284f03f0
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
90 changed files with 630 additions and 331 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
INSTALL.md
View File

@ -15,7 +15,7 @@ was tested against a CentOS 7 x86_64 setup.
Install tools and dependencies used for development: Install tools and dependencies used for development:
# yum -y install git java-11-openjdk java-11-openjdk-devel \ # yum -y install git java-17-openjdk java-17-openjdk-devel \
mysql mysql-server mkisofs git gcc python MySQL-python openssh-clients wget mysql mysql-server mkisofs git gcc python MySQL-python openssh-clients wget
Set up Maven (3.6.0): Set up Maven (3.6.0):

6
debian/control vendored
View File

@ -17,14 +17,14 @@ Description: A common package which contains files which are shared by several C
Package: cloudstack-management Package: cloudstack-management
Architecture: all Architecture: all
Depends: ${python3:Depends}, openjdk-11-jre-headless | java11-runtime-headless | java11-runtime | openjdk-11-jre-headless | zulu-11, cloudstack-common (= ${source:Version}), net-tools, sudo, python3-mysql.connector, augeas-tools, mysql-client | mariadb-client, adduser, bzip2, ipmitool, file, gawk, iproute2, qemu-utils, rng-tools, python3-dnspython, lsb-release, init-system-helpers (>= 1.14~), python3-setuptools Depends: ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), net-tools, sudo, python3-mysql.connector, augeas-tools, mysql-client | mariadb-client, adduser, bzip2, ipmitool, file, gawk, iproute2, qemu-utils, rng-tools, python3-dnspython, lsb-release, init-system-helpers (>= 1.14~), python3-setuptools
Conflicts: cloud-server, cloud-client, cloud-client-ui Conflicts: cloud-server, cloud-client, cloud-client-ui
Description: CloudStack server library Description: CloudStack server library
The CloudStack management server The CloudStack management server
Package: cloudstack-agent Package: cloudstack-agent
Architecture: all Architecture: all
Depends: ${python:Depends}, ${python3:Depends}, openjdk-11-jre-headless | java11-runtime-headless | java11-runtime | openjdk-11-jre-headless | zulu-11, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, lsb-release, ufw, apparmor Depends: ${python:Depends}, ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, lsb-release, ufw, apparmor
Recommends: init-system-helpers Recommends: init-system-helpers
Conflicts: cloud-agent, cloud-agent-libs, cloud-agent-deps, cloud-agent-scripts Conflicts: cloud-agent, cloud-agent-libs, cloud-agent-deps, cloud-agent-scripts
Description: CloudStack agent Description: CloudStack agent
@ -34,7 +34,7 @@ Description: CloudStack agent
Package: cloudstack-usage Package: cloudstack-usage
Architecture: all Architecture: all
Depends: openjdk-11-jre-headless | java11-runtime-headless | java11-runtime | openjdk-11-jre-headless | zulu-11, cloudstack-common (= ${source:Version}), init-system-helpers Depends: openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), init-system-helpers
Description: CloudStack usage monitor Description: CloudStack usage monitor
The CloudStack usage monitor provides usage accounting across the entire cloud for The CloudStack usage monitor provides usage accounting across the entire cloud for
cloud operators to charge based on usage parameters. cloud operators to charge based on usage parameters.

2
engine/api/pom.xml
View File

@ -71,7 +71,7 @@
<dependency> <dependency>
<groupId>com.sun.xml.bind</groupId> <groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId> <artifactId>jaxb-impl</artifactId>
<version>${cs.jaxb.version}</version> <version>${cs.jaxb.impl.version}</version>
</dependency> </dependency>
</dependencies> </dependencies>
</project> </project>

6
engine/schema/pom.xml
View File

@ -101,8 +101,10 @@
for (template in templateList) { for (template in templateList) {
def data = lines.findAll { it.contains(template) } def data = lines.findAll { it.contains(template) }
if (data != null) { if (data != null) {
def hypervisor = template.tokenize('-')[-1] if (data.size() > 0) {
pom.properties["$hypervisor" + ".checksum"] = data[0].tokenize(' ')[0] def hypervisor = template.tokenize('-')[-1]
pom.properties["$hypervisor" + ".checksum"] = data[0].tokenize(' ')[0]
}
} }
} }
</source> </source>

6
engine/schema/src/main/resources/META-INF/db/schema-41900to42000.sql
View File

@ -28,3 +28,9 @@ DROP INDEX `i_resource_count__type_accountId`,
DROP INDEX `i_resource_count__type_domaintId`, DROP INDEX `i_resource_count__type_domaintId`,
ADD UNIQUE INDEX `i_resource_count__type_tag_accountId` (`type`,`tag`,`account_id`), ADD UNIQUE INDEX `i_resource_count__type_tag_accountId` (`type`,`tag`,`account_id`),
ADD UNIQUE INDEX `i_resource_count__type_tag_domaintId` (`type`,`tag`,`domain_id`); ADD UNIQUE INDEX `i_resource_count__type_tag_domaintId` (`type`,`tag`,`domain_id`);
-- Update Default System offering for Router to 512MiB
UPDATE `cloud`.`service_offering` SET ram_size = 512 WHERE unique_name IN ("Cloud.Com-SoftwareRouter", "Cloud.Com-SoftwareRouter-Local",
"Cloud.Com-InternalLBVm", "Cloud.Com-InternalLBVm-Local",
"Cloud.Com-ElasticLBVm", "Cloud.Com-ElasticLBVm-Local")
AND system_use = 1 AND ram_size < 512;

2
framework/rest/pom.xml
View File

@ -68,7 +68,7 @@
<dependency> <dependency>
<groupId>com.sun.xml.bind</groupId> <groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId> <artifactId>jaxb-impl</artifactId>
<version>${cs.jaxb.version}</version> <version>${cs.jaxb.impl.version}</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.apache.cxf</groupId> <groupId>org.apache.cxf</groupId>

10
packaging/centos8/cloud.spec
View File

@ -52,7 +52,7 @@ intelligent IaaS cloud implementation.
%package management %package management
Summary: CloudStack management server UI Summary: CloudStack management server UI
Requires: java-11-openjdk Requires: java-17-openjdk
Requires: (tzdata-java or timezone-java) Requires: (tzdata-java or timezone-java)
Requires: python3 Requires: python3
Requires: bash Requires: bash
@ -98,7 +98,7 @@ The Apache CloudStack files shared between agent and management server
%package agent %package agent
Summary: CloudStack Agent for KVM hypervisors Summary: CloudStack Agent for KVM hypervisors
Requires: (openssh-clients or openssh) Requires: (openssh-clients or openssh)
Requires: java-11-openjdk Requires: java-17-openjdk
Requires: tzdata-java Requires: tzdata-java
Requires: %{name}-common = %{_ver} Requires: %{name}-common = %{_ver}
Requires: libvirt Requires: libvirt
@ -135,7 +135,7 @@ The CloudStack baremetal agent
%package usage %package usage
Summary: CloudStack Usage calculation server Summary: CloudStack Usage calculation server
Requires: java-11-openjdk Requires: java-17-openjdk
Requires: tzdata-java Requires: tzdata-java
Group: System Environment/Libraries Group: System Environment/Libraries
%description usage %description usage
@ -556,8 +556,8 @@ if [ -f "/usr/share/cloudstack-common/scripts/installer/cloudstack-help-text" ];
fi fi
%post marvin %post marvin
pip install --upgrade https://files.pythonhosted.org/packages/08/1f/42d74bae9dd6dcfec67c9ed0f3fa482b1ae5ac5f117ca82ab589ecb3ca19/mysql_connector_python-8.0.31-py2.py3-none-any.whl pip3 install --upgrade https://files.pythonhosted.org/packages/08/1f/42d74bae9dd6dcfec67c9ed0f3fa482b1ae5ac5f117ca82ab589ecb3ca19/mysql_connector_python-8.0.31-py2.py3-none-any.whl
pip install --upgrade /usr/share/cloudstack-marvin/Marvin-*.tar.gz pip3 install --upgrade /usr/share/cloudstack-marvin/Marvin-*.tar.gz
#No default permission as the permission setup is complex #No default permission as the permission setup is complex
%files management %files management

2
packaging/systemd/cloudstack-management.default
View File

@ -15,7 +15,7 @@
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
JAVA_OPTS="-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx2G -XX:+UseParallelGC -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:ErrorFile=/var/log/cloudstack/management/cloudstack-management.err " JAVA_OPTS="-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx2G -XX:+UseParallelGC -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:ErrorFile=/var/log/cloudstack/management/cloudstack-management.err --add-opens=java.base/java.lang=ALL-UNNAMED --add-exports=java.base/sun.security.x509=ALL-UNNAMED"
CLASSPATH="/usr/share/cloudstack-management/lib/*:/etc/cloudstack/management:/usr/share/cloudstack-common:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management:/usr/share/java/mysql-connector-java.jar:/usr/share/cloudstack-mysql-ha/lib/*" CLASSPATH="/usr/share/cloudstack-management/lib/*:/etc/cloudstack/management:/usr/share/cloudstack-common:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management:/usr/share/java/mysql-connector-java.jar:/usr/share/cloudstack-mysql-ha/lib/*"

2
packaging/systemd/cloudstack-usage.default
View File

@ -15,7 +15,7 @@
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
JAVA_OPTS="-Xms256m -Xmx2048m" JAVA_OPTS="-Xms256m -Xmx2048m --add-opens=java.base/java.lang=ALL-UNNAMED"
CLASSPATH="/usr/share/cloudstack-usage/*:/usr/share/cloudstack-usage/lib/*:/usr/share/cloudstack-mysql-ha/lib/*:/etc/cloudstack/usage:/usr/share/java/mysql-connector-java.jar" CLASSPATH="/usr/share/cloudstack-usage/*:/usr/share/cloudstack-usage/lib/*:/usr/share/cloudstack-mysql-ha/lib/*:/etc/cloudstack/usage:/usr/share/java/mysql-connector-java.jar"

2
plugins/hypervisors/baremetal/pom.xml
View File

@ -45,7 +45,7 @@
<dependency> <dependency>
<groupId>com.sun.xml.bind</groupId> <groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId> <artifactId>jaxb-impl</artifactId>
<version>${cs.jaxb.version}</version> <version>${cs.jaxb.impl.version}</version>
</dependency> </dependency>
</dependencies> </dependencies>
</project> </project>

1
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtPatchSystemVmCommandWrapper.java
View File

@ -77,6 +77,7 @@ public class LibvirtPatchSystemVmCommandWrapper extends CommandWrapper<PatchSyst
if (patchResult.first()) { if (patchResult.first()) {
String scriptVersion = lines[1]; String scriptVersion = lines[1];
if (StringUtils.isNotEmpty(patchResult.second())) { if (StringUtils.isNotEmpty(patchResult.second())) {
logger.debug("Patch result of systemVM {}: {}", sysVMName, patchResult.second());
String res = patchResult.second().replace("\n", " "); String res = patchResult.second().replace("\n", " ");
String[] output = res.split(":"); String[] output = res.split(":");
if (output.length != 2) { if (output.length != 2) {

4
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/version/KubernetesVersionManagerImpl.java
View File

@ -102,7 +102,9 @@ public class KubernetesVersionManagerImpl extends ManagerBase implements Kuberne
if (template != null) { if (template != null) {
response.setIsoId(template.getUuid()); response.setIsoId(template.getUuid());
response.setIsoName(template.getName()); response.setIsoName(template.getName());
response.setIsoState(template.getState().toString()); if (template.getState() != null) {
response.setIsoState(template.getState().toString());
}
response.setDirectDownload(template.isDirectDownload()); response.setDirectDownload(template.isDirectDownload());
} }
response.setCreated(kubernetesSupportedVersion.getCreated()); response.setCreated(kubernetesSupportedVersion.getCreated());

2
plugins/network-elements/internal-loadbalancer/src/main/java/org/apache/cloudstack/network/lb/InternalLoadBalancerVMManager.java
View File

@ -32,7 +32,7 @@ import com.cloud.vm.VirtualMachineProfile.Param;
public interface InternalLoadBalancerVMManager { public interface InternalLoadBalancerVMManager {
//RAM/CPU for the system offering used by Internal LB VMs //RAM/CPU for the system offering used by Internal LB VMs
public static final int DEFAULT_INTERNALLB_VM_RAMSIZE = 256; // 256 MB public static final int DEFAULT_INTERNALLB_VM_RAMSIZE = 512; // 512 MB
public static final int DEFAULT_INTERNALLB_VM_CPU_MHZ = 256; // 256 MHz public static final int DEFAULT_INTERNALLB_VM_CPU_MHZ = 256; // 256 MHz
/** /**

7
pom.xml
View File

@ -50,7 +50,7 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<project.systemvm.template.location>https://download.cloudstack.org/systemvm</project.systemvm.template.location> <project.systemvm.template.location>https://download.cloudstack.org/systemvm</project.systemvm.template.location>
<project.systemvm.template.version>4.19.0.0</project.systemvm.template.version> <project.systemvm.template.version>4.20.0.0</project.systemvm.template.version>
<sonar.organization>apache</sonar.organization> <sonar.organization>apache</sonar.organization>
<sonar.host.url>https://sonarcloud.io</sonar.host.url> <sonar.host.url>https://sonarcloud.io</sonar.host.url>
@ -151,7 +151,9 @@
<cs.maven-javadoc-plugin.version>3.1.1</cs.maven-javadoc-plugin.version> <cs.maven-javadoc-plugin.version>3.1.1</cs.maven-javadoc-plugin.version>
<cs.javax.annotation.version>1.3.2</cs.javax.annotation.version> <cs.javax.annotation.version>1.3.2</cs.javax.annotation.version>
<cs.jaxb.version>2.3.0</cs.jaxb.version> <cs.jaxb.version>2.3.0</cs.jaxb.version>
<cs.jaxws.version>2.3.2-1</cs.jaxws.version> <cs.jaxb.impl.version>2.3.9</cs.jaxb.impl.version>
<cs.jakarta.xml.bind.version>2.3.3</cs.jakarta.xml.bind.version>
<cs.jaxws.version>2.3.7</cs.jaxws.version>
<cs.jersey-client.version>2.26</cs.jersey-client.version> <cs.jersey-client.version>2.26</cs.jersey-client.version>
<cs.jetty.version>9.4.51.v20230217</cs.jetty.version> <cs.jetty.version>9.4.51.v20230217</cs.jetty.version>
<cs.jetty-maven-plugin.version>9.4.27.v20200227</cs.jetty-maven-plugin.version> <cs.jetty-maven-plugin.version>9.4.27.v20200227</cs.jetty-maven-plugin.version>
@ -1040,6 +1042,7 @@
<exclude>systemvm/agent/js/jquery.js</exclude> <exclude>systemvm/agent/js/jquery.js</exclude>
<exclude>systemvm/agent/js/jquery.flot.navigate.js</exclude> <exclude>systemvm/agent/js/jquery.flot.navigate.js</exclude>
<exclude>systemvm/agent/noVNC/**</exclude> <exclude>systemvm/agent/noVNC/**</exclude>
<exclude>systemvm/agent/packages/**</exclude>
<exclude>systemvm/debian/**</exclude> <exclude>systemvm/debian/**</exclude>
<exclude>test/integration/component/test_host_ha.sh</exclude> <exclude>test/integration/component/test_host_ha.sh</exclude>
<exclude>test/systemvm/README.md</exclude> <exclude>test/systemvm/README.md</exclude>

6
scripts/installer/cloudstack-help-text
View File

@ -40,4 +40,10 @@ printf " * Release notes: https://docs.cloudstack.apache.org/en/${ACL_MINO
printf " * Join mailing lists: https://cloudstack.apache.org/mailing-lists.html\n" printf " * Join mailing lists: https://cloudstack.apache.org/mailing-lists.html\n"
printf " * Take the survey: https://cloudstack.apache.org/survey.html\n" printf " * Take the survey: https://cloudstack.apache.org/survey.html\n"
printf " * Report issues: https://github.com/apache/cloudstack/issues/new\n" printf " * Report issues: https://github.com/apache/cloudstack/issues/new\n"
if [ "$1" = "management" ];then
printf "\nSince Apache CloudStack 4.20.0.0, the System VMs and virtual routers require at least 512 MiB memory, please check the System Offerings."
printf "\nMore information can be found at https://docs.cloudstack.apache.org/en/${ACL_MINOR_VERSION:-latest}/upgrading/upgrade/_sysvm_restart.html\n"
fi
printf "\n" printf "\n"

4
server/src/main/java/com/cloud/configuration/Config.java
View File

@ -959,7 +959,7 @@ public enum Config {
ManagementServer.class, ManagementServer.class,
Integer.class, Integer.class,
"network.loadbalancer.basiczone.elb.vm.ram.size", "network.loadbalancer.basiczone.elb.vm.ram.size",
"128", "512",
"Memory in MB for the elastic load balancer vm", "Memory in MB for the elastic load balancer vm",
null), null),
ElasticLoadBalancerVmCpuMhz( ElasticLoadBalancerVmCpuMhz(
@ -1291,7 +1291,7 @@ public enum Config {
"The allowable clock difference in milliseconds between when an SSO login request is made and when it is received.", "The allowable clock difference in milliseconds between when an SSO login request is made and when it is received.",
null), null),
//NetworkType("Hidden", ManagementServer.class, String.class, "network.type", "vlan", "The type of network that this deployment will use.", "vlan,direct"), //NetworkType("Hidden", ManagementServer.class, String.class, "network.type", "vlan", "The type of network that this deployment will use.", "vlan,direct"),
RouterRamSize("Hidden", NetworkOrchestrationService.class, Integer.class, "router.ram.size", "256", "Default RAM for router VM (in MB).", null), RouterRamSize("Hidden", NetworkOrchestrationService.class, Integer.class, "router.ram.size", "512", "Default RAM for router VM (in MB).", null),
DefaultPageSize("Advanced", ManagementServer.class, Long.class, "default.page.size", "500", "Default page size for API list* commands", null), DefaultPageSize("Advanced", ManagementServer.class, Long.class, "default.page.size", "500", "Default page size for API list* commands", null),

2
systemvm/agent/noVNC/keymaps/generate-language-keymaps.py
View File

@ -95,7 +95,7 @@ def generate_js_file(keymap_file):
js_config.append(" * layout : %s\n" % layout) js_config.append(" * layout : %s\n" % layout)
js_config.append(" */\n") js_config.append(" */\n")
js_config.append("export default {\n") js_config.append("export default {\n")
for keycode in dict(sorted(result_mappings.items(), key=lambda item: int(item[0]))): for keycode in dict(sorted(list(result_mappings.items()), key=lambda item: int(item[0]))):
js_config.append("%10s : \"%s\",\n" % ("\"" + str(keycode) + "\"", result_mappings[keycode].strip())) js_config.append("%10s : \"%s\",\n" % ("\"" + str(keycode) + "\"", result_mappings[keycode].strip()))
js_config.append("}\n") js_config.append("}\n")
for line in js_config: for line in js_config:

11
systemvm/agent/packages/packages.ini Normal file
View File

@ -0,0 +1,11 @@
[python-is-python3]
debian_os=11
package_name=python-is-python3
file_name=python-is-python3_3.9.2-1_all.deb
conflicted_packages=python-is-python2
[python3-netaddr]
debian_os=11
package_name=python3-netaddr
file_name=python3-netaddr_0.7.19-5_all.deb
conflicted_packages=

BIN
systemvm/agent/packages/python-is-python3_3.9.2-1_all.deb Normal file
View File

Binary file not shown.

BIN
systemvm/agent/packages/python3-netaddr_0.7.19-5_all.deb Normal file
View File

Binary file not shown.

2
systemvm/debian/etc/apache2/vhost.template
View File

@ -93,7 +93,7 @@
# Enable/Disable SSL for this virtual host. # Enable/Disable SSL for this virtual host.
SSLEngine on SSLEngine on
SSLProtocol TLSv1.2 SSLProtocol TLSv1.2
SSLCipherSuite @SECLEVEL=1:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA SSLCipherSuite @SECLEVEL=0:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
SSLHonorCipherOrder on SSLHonorCipherOrder on
# A self-signed (snakeoil) certificate can be created by installing # A self-signed (snakeoil) certificate can be created by installing

4
systemvm/debian/opt/cloud/bin/baremetal-vr.py
View File

@ -59,8 +59,8 @@ class ShellCmd(object):
err = [] err = []
err.append('failed to execute shell command: %s' % self.cmd) err.append('failed to execute shell command: %s' % self.cmd)
err.append('return code: %s' % self.process.returncode) err.append('return code: %s' % self.process.returncode)
err.append('stdout: %s' % self.stdout) err.append('stdout: %s' % self.stdout.decode())
err.append('stderr: %s' % self.stderr) err.append('stderr: %s' % self.stderr.decode())
raise Exception('\n'.join(err)) raise Exception('\n'.join(err))
self.return_code = self.process.returncode self.return_code = self.process.returncode

106
systemvm/debian/opt/cloud/bin/configure.py
View File

@ -21,8 +21,9 @@ import logging
import os import os
import re import re
import sys import sys
import urllib import urllib.request
import urllib2 import urllib.parse
import urllib.error
import time import time
import copy import copy
@ -41,9 +42,12 @@ from cs.CsProcess import CsProcess
from cs.CsStaticRoutes import CsStaticRoutes from cs.CsStaticRoutes import CsStaticRoutes
from cs.CsVpcGuestNetwork import CsVpcGuestNetwork from cs.CsVpcGuestNetwork import CsVpcGuestNetwork
ICMPV6_TYPE_ANY = "{ destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, mld-listener-query, mld-listener-report, mld-listener-done, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, nd-redirect, router-renumbering }" ICMPV6_TYPE_ANY = "{ destination-unreachable, packet-too-big, time-exceeded, parameter-problem, \
echo-request, echo-reply, mld-listener-query, mld-listener-report, mld-listener-done, \
nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, nd-redirect, router-renumbering }"
TCP_UDP_PORT_ANY = "{ 0-65535 }" TCP_UDP_PORT_ANY = "{ 0-65535 }"
def removeUndesiredCidrs(cidrs, version): def removeUndesiredCidrs(cidrs, version):
version_char = ":" version_char = ":"
if version == 4: if version == 4:
@ -61,15 +65,17 @@ def removeUndesiredCidrs(cidrs, version):
return cidrs return cidrs
return None return None
def appendStringIfNotEmpty(s1, s2): def appendStringIfNotEmpty(s1, s2):
if s2: if s2:
if type(s2) != str: if not isinstance(s2, str):
s2 = str(s2) s2 = str(s2)
if s1: if s1:
return s1 + " " + s2 return s1 + " " + s2
return s2 return s2
return s1 return s1
class CsPassword(CsDataBag): class CsPassword(CsDataBag):
TOKEN_FILE = "/tmp/passwdsrvrtoken" TOKEN_FILE = "/tmp/passwdsrvrtoken"
@ -107,10 +113,10 @@ class CsPassword(CsDataBag):
if proc.find(): if proc.find():
url = "http://%s:8080/" % server_ip url = "http://%s:8080/" % server_ip
payload = {"ip": vm_ip, "password": password, "token": token} payload = {"ip": vm_ip, "password": password, "token": token}
data = urllib.urlencode(payload) data = urllib.parse.urlencode(payload).encode()
request = urllib2.Request(url, data=data, headers={"DomU_Request": "save_password"}) request = urllib.request.Request(url, data=data, headers={"DomU_Request": "save_password"})
try: try:
resp = urllib2.urlopen(request, data) resp = urllib.request.urlopen(request, data)
logging.debug("Update password server result: http:%s, content:%s" % (resp.code, resp.read())) logging.debug("Update password server result: http:%s, content:%s" % (resp.code, resp.read()))
except Exception as e: except Exception as e:
logging.error("Failed to update password server due to: %s" % e) logging.error("Failed to update password server due to: %s" % e)
@ -165,15 +171,15 @@ class CsAcl(CsDataBag):
icmp_type = '' icmp_type = ''
rule = self.rule rule = self.rule
icmp_type = "any" icmp_type = "any"
if "icmp_type" in self.rule.keys() and self.rule['icmp_type'] != -1: if "icmp_type" in list(self.rule.keys()) and self.rule['icmp_type'] != -1:
icmp_type = self.rule['icmp_type'] icmp_type = self.rule['icmp_type']
if "icmp_code" in self.rule.keys() and rule['icmp_code'] != -1: if "icmp_code" in list(self.rule.keys()) and rule['icmp_code'] != -1:
icmp_type = "%s/%s" % (self.rule['icmp_type'], self.rule['icmp_code']) icmp_type = "%s/%s" % (self.rule['icmp_type'], self.rule['icmp_code'])
rnge = '' rnge = ''
if "first_port" in self.rule.keys() and \ if "first_port" in list(self.rule.keys()) and \
self.rule['first_port'] == self.rule['last_port']: self.rule['first_port'] == self.rule['last_port']:
rnge = " --dport %s " % self.rule['first_port'] rnge = " --dport %s " % self.rule['first_port']
if "first_port" in self.rule.keys() and \ if "first_port" in list(self.rule.keys()) and \
self.rule['first_port'] != self.rule['last_port']: self.rule['first_port'] != self.rule['last_port']:
rnge = " --dport %s:%s" % (rule['first_port'], rule['last_port']) rnge = " --dport %s:%s" % (rule['first_port'], rule['last_port'])
@ -278,14 +284,14 @@ class CsAcl(CsDataBag):
self.device = obj['device'] self.device = obj['device']
self.ip = obj['nic_ip'] self.ip = obj['nic_ip']
self.ip6_cidr = None self.ip6_cidr = None
if "nic_ip6_cidr" in obj.keys(): if "nic_ip6_cidr" in list(obj.keys()):
self.ip6_cidr = obj['nic_ip6_cidr'] self.ip6_cidr = obj['nic_ip6_cidr']
self.netmask = obj['nic_netmask'] self.netmask = obj['nic_netmask']
self.config = config self.config = config
self.cidr = "%s/%s" % (self.ip, self.netmask) self.cidr = "%s/%s" % (self.ip, self.netmask)
if "ingress_rules" in obj.keys(): if "ingress_rules" in list(obj.keys()):
self.ingress = obj['ingress_rules'] self.ingress = obj['ingress_rules']
if "egress_rules" in obj.keys(): if "egress_rules" in list(obj.keys()):
self.egress = obj['egress_rules'] self.egress = obj['egress_rules']
self.fw = config.get_fw() self.fw = config.get_fw()
self.ipv6_acl = config.get_ipv6_acl() self.ipv6_acl = config.get_ipv6_acl()
@ -308,9 +314,9 @@ class CsAcl(CsDataBag):
self.ipv6_acl.insert(0, {'type': "chain", 'chain': chain}) self.ipv6_acl.insert(0, {'type': "chain", 'chain': chain})
for rule in rule_list: for rule in rule_list:
cidr = rule['cidr'] cidr = rule['cidr']
if cidr != None and cidr != "": if cidr is not None and cidr != "":
cidr = removeUndesiredCidrs(cidr, 4) cidr = removeUndesiredCidrs(cidr, 4)
if cidr == None or cidr == "": if cidr is None or cidr == "":
continue continue
addr = "" addr = ""
if cidr: if cidr:
@ -352,7 +358,7 @@ class CsAcl(CsDataBag):
proto = "%s dport %s" % (proto, port) proto = "%s dport %s" % (proto, port)
action = "drop" action = "drop"
if 'allowed' in rule.keys() and rule['allowed']: if 'allowed' in list(rule.keys()) and rule['allowed']:
action = "accept" action = "accept"
rstr = addr rstr = addr
@ -376,9 +382,9 @@ class CsAcl(CsDataBag):
for i in rule_list: for i in rule_list:
ruleData = copy.copy(i) ruleData = copy.copy(i)
cidr = ruleData['cidr'] cidr = ruleData['cidr']
if cidr != None and cidr != "": if cidr is not None and cidr != "":
cidr = removeUndesiredCidrs(cidr, 6) cidr = removeUndesiredCidrs(cidr, 6)
if cidr == None or cidr == "": if cidr is None or cidr == "":
continue continue
ruleData['cidr'] = cidr ruleData['cidr'] = cidr
r = self.AclRule(direction, self, ruleData, self.config, count) r = self.AclRule(direction, self, ruleData, self.config, count)
@ -411,9 +417,9 @@ class CsAcl(CsDataBag):
self.type = rule['type'] self.type = rule['type']
self.icmp_type = "any" self.icmp_type = "any"
self.protocol = self.type self.protocol = self.type
if "icmp_type" in rule.keys() and rule['icmp_type'] != -1: if "icmp_type" in list(rule.keys()) and rule['icmp_type'] != -1:
self.icmp_type = rule['icmp_type'] self.icmp_type = rule['icmp_type']
if "icmp_code" in rule.keys() and rule['icmp_code'] != -1: if "icmp_code" in list(rule.keys()) and rule['icmp_code'] != -1:
self.icmp_type = "%s/%s" % (self.icmp_type, rule['icmp_code']) self.icmp_type = "%s/%s" % (self.icmp_type, rule['icmp_code'])
if self.type == "protocol": if self.type == "protocol":
if rule['protocol'] == 41: if rule['protocol'] == 41:
@ -421,11 +427,11 @@ class CsAcl(CsDataBag):
self.protocol = rule['protocol'] self.protocol = rule['protocol']
self.action = "DROP" self.action = "DROP"
self.dport = "" self.dport = ""
if 'allowed' in rule.keys() and rule['allowed']: if 'allowed' in list(rule.keys()) and rule['allowed']:
self.action = "ACCEPT" self.action = "ACCEPT"
if 'first_port' in rule.keys(): if 'first_port' in list(rule.keys()):
self.dport = "-m %s --dport %s" % (self.protocol, rule['first_port']) self.dport = "-m %s --dport %s" % (self.protocol, rule['first_port'])
if 'last_port' in rule.keys() and self.dport and \ if 'last_port' in list(rule.keys()) and self.dport and \
rule['last_port'] != rule['first_port']: rule['last_port'] != rule['first_port']:
self.dport = "%s:%s" % (self.dport, rule['last_port']) self.dport = "%s:%s" % (self.dport, rule['last_port'])
@ -488,7 +494,7 @@ class CsIpv6Firewall(CsDataBag):
continue continue
rule = self.dbag[item] rule = self.dbag[item]
if chains_added == False: if chains_added is False:
guest_cidr = rule['guest_ip6_cidr'] guest_cidr = rule['guest_ip6_cidr']
parent_chain = "fw_forward" parent_chain = "fw_forward"
chain = "fw_chain_egress" chain = "fw_chain_egress"
@ -640,23 +646,26 @@ class CsVmMetadata(CsDataBag):
fh = open(dest, "w") fh = open(dest, "w")
self.__exflock(fh) self.__exflock(fh)
if data is not None: if data is not None:
fh.write(data) if isinstance(data, str):
fh.write(data)
elif isinstance(data, bytes):
fh.write(data.decode())
else: else:
fh.write("") fh.write("")
self.__unflock(fh) self.__unflock(fh)
fh.close() fh.close()
os.chmod(dest, 0644) os.chmod(dest, 0o644)
if folder == "metadata" or folder == "meta-data": if folder == "metadata" or folder == "meta-data":
try: try:
os.makedirs(metamanifestdir, 0755) os.makedirs(metamanifestdir, 0o755)
except OSError as e: except OSError as e:
# error 17 is already exists, we do it this way for concurrency # error 17 is already exists, we do it this way for concurrency
if e.errno != 17: if e.errno != 17:
print "failed to make directories " + metamanifestdir + " due to :" + e.strerror print("failed to make directories " + metamanifestdir + " due to :" + e.strerror)
sys.exit(1) sys.exit(1)
if os.path.exists(metamanifest): if os.path.exists(metamanifest):
fh = open(metamanifest, "r+a") fh = open(metamanifest, "a+")
self.__exflock(fh) self.__exflock(fh)
if file not in fh.read(): if file not in fh.read():
fh.write(file + '\n') fh.write(file + '\n')
@ -670,17 +679,17 @@ class CsVmMetadata(CsDataBag):
fh.close() fh.close()
if os.path.exists(metamanifest): if os.path.exists(metamanifest):
os.chmod(metamanifest, 0644) os.chmod(metamanifest, 0o644)
def __htaccess(self, ip, folder, file): def __htaccess(self, ip, folder, file):
entry = "RewriteRule ^" + file + "$ ../" + folder + "/%{REMOTE_ADDR}/" + file + " [L,NC,QSA]" entry = "RewriteRule ^" + file + "$ ../" + folder + "/%{REMOTE_ADDR}/" + file + " [L,NC,QSA]"
htaccessFolder = "/var/www/html/latest" htaccessFolder = "/var/www/html/latest"
htaccessFile = htaccessFolder + "/.htaccess" htaccessFile = htaccessFolder + "/.htaccess"
CsHelper.mkdir(htaccessFolder, 0755, True) CsHelper.mkdir(htaccessFolder, 0o755, True)
if os.path.exists(htaccessFile): if os.path.exists(htaccessFile):
fh = open(htaccessFile, "r+a") fh = open(htaccessFile, "a+")
self.__exflock(fh) self.__exflock(fh)
if entry not in fh.read(): if entry not in fh.read():
fh.write(entry + '\n') fh.write(entry + '\n')
@ -699,11 +708,11 @@ class CsVmMetadata(CsDataBag):
htaccessFile = htaccessFolder+"/.htaccess" htaccessFile = htaccessFolder+"/.htaccess"
try: try:
os.makedirs(htaccessFolder, 0755) os.makedirs(htaccessFolder, 0o755)
except OSError as e: except OSError as e:
# error 17 is already exists, we do it this way for sake of concurrency # error 17 is already exists, we do it this way for sake of concurrency
if e.errno != 17: if e.errno != 17:
print "failed to make directories " + htaccessFolder + " due to :" + e.strerror print("failed to make directories " + htaccessFolder + " due to :" + e.strerror)
sys.exit(1) sys.exit(1)
fh = open(htaccessFile, "w") fh = open(htaccessFile, "w")
@ -717,7 +726,7 @@ class CsVmMetadata(CsDataBag):
htaccessFolder = "/var/www/html/latest" htaccessFolder = "/var/www/html/latest"
htaccessFile = htaccessFolder + "/.htaccess" htaccessFile = htaccessFolder + "/.htaccess"
fh = open(htaccessFile, "r+a") fh = open(htaccessFile, "a+")
self.__exflock(fh) self.__exflock(fh)
if entry not in fh.read(): if entry not in fh.read():
fh.write(entry + '\n') fh.write(entry + '\n')
@ -734,7 +743,7 @@ class CsVmMetadata(CsDataBag):
try: try:
flock(file, LOCK_EX) flock(file, LOCK_EX)
except IOError as e: except IOError as e:
print "failed to lock file" + file.name + " due to : " + e.strerror print("failed to lock file" + file.name + " due to : " + e.strerror)
sys.exit(1) # FIXME sys.exit(1) # FIXME
return True return True
@ -742,7 +751,7 @@ class CsVmMetadata(CsDataBag):
try: try:
flock(file, LOCK_UN) flock(file, LOCK_UN)
except IOError as e: except IOError as e:
print "failed to unlock file" + file.name + " due to : " + e.strerror print("failed to unlock file" + file.name + " due to : " + e.strerror)
sys.exit(1) # FIXME sys.exit(1) # FIXME
return True return True
@ -838,9 +847,9 @@ class CsSite2SiteVpn(CsDataBag):
file.addeq(" authby=secret") file.addeq(" authby=secret")
file.addeq(" keyexchange=%s" % ikeversion) file.addeq(" keyexchange=%s" % ikeversion)
file.addeq(" ike=%s" % ikepolicy) file.addeq(" ike=%s" % ikepolicy)
file.addeq(" ikelifetime=%s" % self.convert_sec_to_h(obj['ike_lifetime'])) file.addeq(" ikelifetime=%s" % self.convert_sec_to_min(obj['ike_lifetime']))
file.addeq(" esp=%s" % esppolicy) file.addeq(" esp=%s" % esppolicy)
file.addeq(" lifetime=%s" % self.convert_sec_to_h(obj['esp_lifetime'])) file.addeq(" lifetime=%s" % self.convert_sec_to_min(obj['esp_lifetime']))
file.addeq(" keyingtries=2") file.addeq(" keyingtries=2")
file.addeq(" auto=route") file.addeq(" auto=route")
if 'encap' not in obj: if 'encap' not in obj:
@ -868,9 +877,9 @@ class CsSite2SiteVpn(CsDataBag):
# This will load the new config # This will load the new config
CsHelper.execute("ipsec reload") CsHelper.execute("ipsec reload")
os.chmod(vpnsecretsfile, 0400) os.chmod(vpnsecretsfile, 0o400)
for i in xrange(3): for i in range(3):
done = True done = True
for peeridx in range(0, len(peerlistarr)): for peeridx in range(0, len(peerlistarr)):
# Check for the proper connection and subnet # Check for the proper connection and subnet
@ -891,9 +900,9 @@ class CsSite2SiteVpn(CsDataBag):
ipinsubnet = '.'.join(octets) ipinsubnet = '.'.join(octets)
CsHelper.execute("timeout 5 ping -c 3 %s" % ipinsubnet) CsHelper.execute("timeout 5 ping -c 3 %s" % ipinsubnet)
def convert_sec_to_h(self, val): def convert_sec_to_min(self, val):
hrs = int(val) / 3600 mins = int(val / 60)
return "%sh" % hrs return "%sm" % mins
class CsVpnUser(CsDataBag): class CsVpnUser(CsDataBag):
@ -1383,7 +1392,7 @@ def main(argv):
databag_map.pop("guest_network") databag_map.pop("guest_network")
def execDatabag(key, db): def execDatabag(key, db):
if key not in db.keys() or 'executor' not in db[key]: if key not in list(db.keys()) or 'executor' not in db[key]:
logging.warn("Unable to find config or executor(s) for the databag type %s" % key) logging.warn("Unable to find config or executor(s) for the databag type %s" % key)
return return
for executor in db[key]['executor']: for executor in db[key]['executor']:
@ -1397,10 +1406,10 @@ def main(argv):
if json_type == "cmd_line": if json_type == "cmd_line":
logging.debug("cmd_line.json changed. All other files will be processed as well.") logging.debug("cmd_line.json changed. All other files will be processed as well.")
for key in databag_map.keys(): for key in list(databag_map.keys()):
execDatabag(key, databag_map) execDatabag(key, databag_map)
execIptables(config) execIptables(config)
elif json_type in databag_map.keys(): elif json_type in list(databag_map.keys()):
execDatabag(json_type, databag_map) execDatabag(json_type, databag_map)
if databag_map[json_type]['process_iptables']: if databag_map[json_type]['process_iptables']:
execIptables(config) execIptables(config)
@ -1411,5 +1420,6 @@ def main(argv):
red.set() red.set()
return 0 return 0
if __name__ == "__main__": if __name__ == "__main__":
main(sys.argv) main(sys.argv)

30
systemvm/debian/opt/cloud/bin/cs/CsAddress.py
View File

@ -19,11 +19,11 @@ import logging
from netaddr import IPAddress, IPNetwork from netaddr import IPAddress, IPNetwork
import subprocess import subprocess
import time import time
import CsHelper from . import CsHelper
from CsDatabag import CsDataBag from .CsDatabag import CsDataBag
from CsApp import CsApache, CsDnsmasq, CsPasswdSvc from .CsApp import CsApache, CsDnsmasq, CsPasswdSvc
from CsRoute import CsRoute from .CsRoute import CsRoute
from CsRule import CsRule from .CsRule import CsRule
VRRP_TYPES = ['guest'] VRRP_TYPES = ['guest']
@ -321,7 +321,7 @@ class CsIP:
logging.info("Configuring address %s on device %s", self.ip(), self.dev) logging.info("Configuring address %s on device %s", self.ip(), self.dev)
cmd = "ip addr add dev %s %s brd +" % (self.dev, self.ip()) cmd = "ip addr add dev %s %s brd +" % (self.dev, self.ip())
CsHelper.execute(cmd) CsHelper.execute(cmd)
cmd = "ifconfig %s mtu %s" % (self.dev, self.mtu()) cmd = "ifconfig %s mtu %s" % (self.dev, self.mtu())
CsHelper.execute(cmd) CsHelper.execute(cmd)
except Exception as e: except Exception as e:
logging.info("Exception occurred ==> %s" % e) logging.info("Exception occurred ==> %s" % e)
@ -364,7 +364,7 @@ class CsIP:
else: else:
# once we start processing public ip's we need to verify there # once we start processing public ip's we need to verify there
# is a default route and add if needed # is a default route and add if needed
if(self.cl.get_gateway()): if self.cl.get_gateway():
route.add_defaultroute(self.cl.get_gateway()) route.add_defaultroute(self.cl.get_gateway())
if self.config.is_router() and self.cl.get_ip6gateway(): if self.config.is_router() and self.cl.get_ip6gateway():
@ -556,7 +556,7 @@ class CsIP:
"-A POSTROUTING -o %s -j SNAT --to-source %s" % "-A POSTROUTING -o %s -j SNAT --to-source %s" %
(self.dev, self.address['public_ip'])]) (self.dev, self.address['public_ip'])])
if self.get_gateway() == self.get_ip_address(): if self.get_gateway() == self.get_ip_address():
for inf, addresses in self.config.address().dbag.iteritems(): for inf, addresses in self.config.address().dbag.items():
if not inf.startswith("eth"): if not inf.startswith("eth"):
continue continue
for address in addresses: for address in addresses:
@ -625,7 +625,7 @@ class CsIP:
if self.config.is_vpc(): if self.config.is_vpc():
if self.get_type() in ["public"] and "gateway" in self.address and self.address["gateway"] and self.address["gateway"] != "None": if self.get_type() in ["public"] and "gateway" in self.address and self.address["gateway"] and self.address["gateway"] != "None":
route.add_route(self.dev, self.address["gateway"]) route.add_route(self.dev, self.address["gateway"])
for inf, addresses in self.config.address().dbag.iteritems(): for inf, addresses in self.config.address().dbag.items():
if not inf.startswith("eth"): if not inf.startswith("eth"):
continue continue
for address in addresses: for address in addresses:
@ -709,7 +709,7 @@ class CsIP:
self.iplist[cidr] = self.dev self.iplist[cidr] = self.dev
def configured(self): def configured(self):
if self.address['cidr'] in self.iplist.keys(): if self.address['cidr'] in list(self.iplist.keys()):
return True return True
return False return False
@ -738,7 +738,7 @@ class CsIP:
return self.dev return self.dev
def hasIP(self, ip): def hasIP(self, ip):
return ip in self.address.values() return ip in list(self.address.values())
def arpPing(self): def arpPing(self):
cmd = "arping -c 1 -I %s -A -U -s %s %s" % ( cmd = "arping -c 1 -I %s -A -U -s %s %s" % (
@ -749,7 +749,7 @@ class CsIP:
# Delete any ips that are configured but not in the bag # Delete any ips that are configured but not in the bag
def compare(self, bag): def compare(self, bag):
if len(self.iplist) > 0 and (self.dev not in bag.keys() or len(bag[self.dev]) == 0): if len(self.iplist) > 0 and (self.dev not in list(bag.keys()) or len(bag[self.dev]) == 0):
# Remove all IPs on this device # Remove all IPs on this device
logging.info( logging.info(
"Will remove all configured addresses on device %s", self.dev) "Will remove all configured addresses on device %s", self.dev)
@ -760,13 +760,13 @@ class CsIP:
# This condition should not really happen but did :) # This condition should not really happen but did :)
# It means an apache file got orphaned after a guest network address # It means an apache file got orphaned after a guest network address
# was deleted # was deleted
if len(self.iplist) == 0 and (self.dev not in bag.keys() or len(bag[self.dev]) == 0): if len(self.iplist) == 0 and (self.dev not in list(bag.keys()) or len(bag[self.dev]) == 0):
app = CsApache(self) app = CsApache(self)
app.remove() app.remove()
for ip in self.iplist: for ip in self.iplist:
found = False found = False
if self.dev in bag.keys(): if self.dev in list(bag.keys()):
for address in bag[self.dev]: for address in bag[self.dev]:
self.setAddress(address) self.setAddress(address)
if (self.hasIP(ip) or self.is_guest_gateway(address, ip)) and address["add"]: if (self.hasIP(ip) or self.is_guest_gateway(address, ip)) and address["add"]:
@ -799,7 +799,7 @@ class CsIP:
remove = [] remove = []
if ip == "all": if ip == "all":
logging.info("Removing addresses from device %s", self.dev) logging.info("Removing addresses from device %s", self.dev)
remove = self.iplist.keys() remove = list(self.iplist.keys())
else: else:
remove.append(ip) remove.append(ip)
for ip in remove: for ip in remove:

4
systemvm/debian/opt/cloud/bin/cs/CsApp.py
View File

@ -16,8 +16,8 @@
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
import os import os
from CsFile import CsFile from .CsFile import CsFile
import CsHelper from . import CsHelper
class CsApp: class CsApp:

4
systemvm/debian/opt/cloud/bin/cs/CsConfig.py
View File

@ -16,8 +16,8 @@
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
from CsDatabag import CsCmdLine, CsGuestNetwork from .CsDatabag import CsCmdLine, CsGuestNetwork
from CsAddress import CsAddress from .CsAddress import CsAddress
import logging import logging

10
systemvm/debian/opt/cloud/bin/cs/CsDatabag.py
View File

@ -33,7 +33,7 @@ class CsDataBag(object):
self.config = config self.config = config
def dump(self): def dump(self):
print self.dbag print(self.dbag)
def get_bag(self): def get_bag(self):
return self.dbag return self.dbag
@ -151,7 +151,7 @@ class CsCmdLine(CsDataBag):
else: else:
passwd = "%s-%s" % (self.get_vpccidr(), self.get_router_id()) passwd = "%s-%s" % (self.get_vpccidr(), self.get_router_id())
md5 = hashlib.md5() md5 = hashlib.md5()
md5.update(passwd) md5.update(passwd.encode())
return md5.hexdigest() return md5.hexdigest()
def get_gateway(self): def get_gateway(self):
@ -191,7 +191,7 @@ class CsGuestNetwork(CsDataBag):
""" Get guestnetwork config parameters """ """ Get guestnetwork config parameters """
def get_dev_data(self, devname): def get_dev_data(self, devname):
if devname in self.dbag and type(self.dbag[devname]) == list and len(self.dbag[devname]) > 0: if devname in self.dbag and isinstance(self.dbag[devname], list) and len(self.dbag[devname]) > 0:
return self.dbag[devname][0] return self.dbag[devname][0]
return {} return {}
@ -223,7 +223,7 @@ class CsGuestNetwork(CsDataBag):
if devname: if devname:
return self.__get_device_router_ip6prelen(devname) return self.__get_device_router_ip6prelen(devname)
else: else:
for key in self.dbag.keys(): for key in list(self.dbag.keys()):
ip6prelen = self.__get_device_router_ip6prelen(key) ip6prelen = self.__get_device_router_ip6prelen(key)
if ip6prelen: if ip6prelen:
return ip6prelen return ip6prelen
@ -240,7 +240,7 @@ class CsGuestNetwork(CsDataBag):
if devname: if devname:
return self.__get_device_router_ip6gateway(devname) return self.__get_device_router_ip6gateway(devname)
else: else:
for key in self.dbag.keys(): for key in list(self.dbag.keys()):
ip6gateway = self.__get_device_router_ip6gateway(key) ip6gateway = self.__get_device_router_ip6gateway(key)
if ip6gateway: if ip6gateway:
return ip6gateway return ip6gateway

4
systemvm/debian/opt/cloud/bin/cs/CsDhcp.py
View File

@ -14,13 +14,13 @@
# KIND, either express or implied. See the License for the # KIND, either express or implied. See the License for the
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
import CsHelper from . import CsHelper
import logging import logging
import os import os
from netaddr import * from netaddr import *
from random import randint from random import randint
import json import json
from CsGuestNetwork import CsGuestNetwork from .CsGuestNetwork import CsGuestNetwork
from cs.CsDatabag import CsDataBag from cs.CsDatabag import CsDataBag
from cs.CsFile import CsFile from cs.CsFile import CsFile
from cs.CsAddress import CsIP from cs.CsAddress import CsIP

4
systemvm/debian/opt/cloud/bin/cs/CsFile.py
View File

@ -70,7 +70,7 @@ class CsFile:
def dump(self): def dump(self):
for line in self.new_config: for line in self.new_config:
print line print(line)
def addeq(self, string): def addeq(self, string):
""" Update a line in a file of the form token=something """ Update a line in a file of the form token=something
@ -153,7 +153,7 @@ class CsFile:
logging.debug("Searching for %s string " % search) logging.debug("Searching for %s string " % search)
for index, line in enumerate(self.new_config): for index, line in enumerate(self.new_config):
print ' line = ' + line print(' line = ' + line)
if line.lstrip().startswith(ignoreLinesStartWith): if line.lstrip().startswith(ignoreLinesStartWith):
continue continue
if search in line: if search in line:

4
systemvm/debian/opt/cloud/bin/cs/CsGuestNetwork.py
View File

@ -15,7 +15,7 @@
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
from merge import DataBag from merge import DataBag
import CsHelper from . import CsHelper
class CsGuestNetwork: class CsGuestNetwork:
@ -27,7 +27,7 @@ class CsGuestNetwork:
db.load() db.load()
dbag = db.getDataBag() dbag = db.getDataBag()
self.config = config self.config = config
if device in dbag.keys() and len(dbag[device]) != 0: if device in list(dbag.keys()) and len(dbag[device]) != 0:
self.data = dbag[device][0] self.data = dbag[device][0]
else: else:
self.guest = False self.guest = False

10
systemvm/debian/opt/cloud/bin/cs/CsHelper.py
View File

@ -87,7 +87,7 @@ def mkdir(name, mode, fatal):
except OSError as e: except OSError as e:
if e.errno != 17: if e.errno != 17:
print("failed to make directories " + name + " due to :" + e.strerror) print("failed to make directories " + name + " due to :" + e.strerror)
if(fatal): if fatal:
sys.exit(1) sys.exit(1)
@ -115,8 +115,8 @@ def get_device_info():
list = [] list = []
for i in execute("ip addr show |grep -v secondary"): for i in execute("ip addr show |grep -v secondary"):
vals = i.strip().lstrip().rstrip().split() vals = i.strip().lstrip().rstrip().split()
if re.search('[0-9]:',vals[0]): if re.search('[0-9]:', vals[0]):
to={} to = {}
to['mtu'] = vals[4] to['mtu'] = vals[4]
list.append(to) list.append(to)
@ -124,7 +124,7 @@ def get_device_info():
if len(list) > 0: if len(list) > 0:
to = list.pop(len(list)-1) to = list.pop(len(list)-1)
else: else:
to={} to = {}
to['ip'] = vals[1] to['ip'] = vals[1]
to['dev'] = vals[-1] to['dev'] = vals[-1]
to['network'] = IPNetwork(to['ip']) to['network'] = IPNetwork(to['ip'])
@ -198,7 +198,7 @@ def execute(command):
returncode = 0 returncode = 0
logging.debug("Command [%s] has the result [%s]" % (command, result)) logging.debug("Command [%s] has the result [%s]" % (command, result))
return result.splitlines() return result.decode().splitlines()
except subprocess.CalledProcessError as e: except subprocess.CalledProcessError as e:
logging.error(e) logging.error(e)
returncode = e.returncode returncode = e.returncode

10
systemvm/debian/opt/cloud/bin/cs/CsLoadBalancer.py
View File

@ -18,9 +18,9 @@ import logging
import os.path import os.path
import re import re
from cs.CsDatabag import CsDataBag from cs.CsDatabag import CsDataBag
from CsProcess import CsProcess from .CsProcess import CsProcess
from CsFile import CsFile from .CsFile import CsFile
import CsHelper from . import CsHelper
HAPROXY_CONF_T = "/etc/haproxy/haproxy.cfg.new" HAPROXY_CONF_T = "/etc/haproxy/haproxy.cfg.new"
HAPROXY_CONF_P = "/etc/haproxy/haproxy.cfg" HAPROXY_CONF_P = "/etc/haproxy/haproxy.cfg"
@ -30,9 +30,9 @@ class CsLoadBalancer(CsDataBag):
""" Manage Load Balancer entries """ """ Manage Load Balancer entries """
def process(self): def process(self):
if "config" not in self.dbag.keys(): if "config" not in list(self.dbag.keys()):
return return
if 'configuration' not in self.dbag['config'][0].keys(): if 'configuration' not in list(self.dbag['config'][0].keys()):
return return
config = self.dbag['config'][0]['configuration'] config = self.dbag['config'][0]['configuration']
file1 = CsFile(HAPROXY_CONF_T) file1 = CsFile(HAPROXY_CONF_T)

2
systemvm/debian/opt/cloud/bin/cs/CsMonitor.py
View File

@ -16,7 +16,7 @@
# under the License. # under the License.
import logging import logging
from cs.CsDatabag import CsDataBag from cs.CsDatabag import CsDataBag
from CsFile import CsFile from .CsFile import CsFile
import json import json
MON_CONFIG = "/etc/monitor.conf" MON_CONFIG = "/etc/monitor.conf"

47
systemvm/debian/opt/cloud/bin/cs/CsNetfilter.py
View File

@ -15,8 +15,8 @@
# KIND, either express or implied. See the License for the # KIND, either express or implied. See the License for the
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
import CsHelper from . import CsHelper
from CsDatabag import CsCmdLine from .CsDatabag import CsCmdLine
import logging import logging
@ -28,7 +28,7 @@ class CsChain(object):
self.count = {} self.count = {}
def add(self, table, chain): def add(self, table, chain):
if table not in self.chain.keys(): if table not in list(self.chain.keys()):
self.chain.setdefault(table, []).append(chain) self.chain.setdefault(table, []).append(chain)
else: else:
self.chain[table].append(chain) self.chain[table].append(chain)
@ -40,7 +40,7 @@ class CsChain(object):
self.count[chain] += 1 self.count[chain] += 1
def get(self, table): def get(self, table):
if table not in self.chain.keys(): if table not in list(self.chain.keys()):
return {} return {}
return self.chain[table] return self.chain[table]
@ -51,7 +51,7 @@ class CsChain(object):
return self.last_added return self.last_added
def has_chain(self, table, chain): def has_chain(self, table, chain):
if table not in self.chain.keys(): if table not in list(self.chain.keys()):
return False return False
if chain not in self.chain[table]: if chain not in self.chain[table]:
return False return False
@ -179,7 +179,7 @@ class CsNetfilters(object):
# For now raising the log. # For now raising the log.
# TODO: Need to fix in the framework. # TODO: Need to fix in the framework.
if ret.returncode != 0: if ret.returncode != 0:
error = ret.communicate()[0] error = ret.communicate()[0].decode()
logging.debug("iptables command got failed ... continuing") logging.debug("iptables command got failed ... continuing")
ruleSet.add(tupledFw) ruleSet.add(tupledFw)
self.chain.add_rule(rule_chain) self.chain.add_rule(rule_chain)
@ -223,14 +223,15 @@ class CsNetfilters(object):
self.rules[:] = [x for x in self.rules if not x == rule] self.rules[:] = [x for x in self.rules if not x == rule]
def add_ip6_chain(self, address_family, table, chain, hook, action): def add_ip6_chain(self, address_family, table, chain, hook, action):
chain_policy = "" chain_policy = ""
if hook: if hook:
chain_policy = "type filter hook %s priority 0;" % hook chain_policy = "type filter hook %s priority 0;" % hook
if chain_policy and action: if chain_policy and action:
chain_policy = "%s policy %s;" % (chain_policy, action) chain_policy = "%s policy %s;" % (chain_policy, action)
CsHelper.execute("nft add chain %s %s %s '{ %s }'" % (address_family, table, chain, chain_policy)) CsHelper.execute("nft add chain %s %s %s '{ %s }'" % (address_family, table, chain, chain_policy))
if hook == "input" or hook == "output": if hook == "input" or hook == "output":
CsHelper.execute("nft add rule %s %s %s icmpv6 type { echo-request, echo-reply, nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept" % (address_family, table, chain)) CsHelper.execute("nft add rule %s %s %s icmpv6 type { echo-request, echo-reply, \
nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept" % (address_family, table, chain))
def apply_ip6_rules(self, rules, type): def apply_ip6_rules(self, rules, type):
if len(rules) == 0: if len(rules) == 0:
@ -238,14 +239,14 @@ class CsNetfilters(object):
address_family = 'ip6' address_family = 'ip6'
table = 'ip6_firewall' table = 'ip6_firewall'
default_chains = [ default_chains = [
{ "chain": "fw_input", "hook": "input", "action": "drop"}, {"chain": "fw_input", "hook": "input", "action": "drop"},
{ "chain": "fw_forward", "hook": "forward", "action": "accept"} {"chain": "fw_forward", "hook": "forward", "action": "accept"}
] ]
if type == "acl": if type == "acl":
table = 'ip6_acl' table = 'ip6_acl'
default_chains = [ default_chains = [
{ "chain": "acl_input", "hook": "input", "action": "drop" }, {"chain": "acl_input", "hook": "input", "action": "drop"},
{ "chain": "acl_forward", "hook": "forward", "action": "accept"} {"chain": "acl_forward", "hook": "forward", "action": "accept"}
] ]
CsHelper.execute("nft add table %s %s" % (address_family, table)) CsHelper.execute("nft add table %s %s" % (address_family, table))
for chain in default_chains: for chain in default_chains:
@ -287,7 +288,7 @@ class CsNetfilter(object):
self.seen = True self.seen = True
def __convert_to_dict(self, rule): def __convert_to_dict(self, rule):
rule = unicode(rule.lstrip()) rule = str(rule.lstrip())
rule = rule.replace('! -', '!_-') rule = rule.replace('! -', '!_-')
rule = rule.replace('-p all', '') rule = rule.replace('-p all', '')
rule = rule.replace(' ', ' ') rule = rule.replace(' ', ' ')
@ -298,8 +299,8 @@ class CsNetfilter(object):
rule = rule.replace('-m state', '-m2 state') rule = rule.replace('-m state', '-m2 state')
rule = rule.replace('ESTABLISHED,RELATED', 'RELATED,ESTABLISHED') rule = rule.replace('ESTABLISHED,RELATED', 'RELATED,ESTABLISHED')
bits = rule.split(' ') bits = rule.split(' ')
rule = dict(zip(bits[0::2], bits[1::2])) rule = dict(list(zip(bits[0::2], bits[1::2])))
if "-A" in rule.keys(): if "-A" in list(rule.keys()):
self.chain = rule["-A"] self.chain = rule["-A"]
return rule return rule
@ -334,7 +335,7 @@ class CsNetfilter(object):
'--to-source', '--to-destination', '--mark'] '--to-source', '--to-destination', '--mark']
str = '' str = ''
for k in order: for k in order:
if k in self.rule.keys(): if k in list(self.rule.keys()):
printable = k.replace('-m2', '-m') printable = k.replace('-m2', '-m')
printable = printable.replace('!_-', '! -') printable = printable.replace('!_-', '! -')
if delete: if delete:
@ -351,7 +352,7 @@ class CsNetfilter(object):
return False return False
if rule.get_chain() != self.get_chain(): if rule.get_chain() != self.get_chain():
return False return False
if len(rule.get_rule().items()) != len(self.get_rule().items()): if len(list(rule.get_rule().items())) != len(list(self.get_rule().items())):
return False return False
common = set(rule.get_rule().items()) & set(self.get_rule().items()) common = set(rule.get_rule().items()) & set(self.get_rule().items())
if len(common) != len(rule.get_rule()): if len(common) != len(rule.get_rule()):

2
systemvm/debian/opt/cloud/bin/cs/CsProcess.py
View File

@ -17,7 +17,7 @@
# under the License. # under the License.
import os import os
import re import re
import CsHelper from . import CsHelper
import logging import logging

19
systemvm/debian/opt/cloud/bin/cs/CsRedundant.py
View File

@ -32,13 +32,13 @@
# -------------------------------------------------------------------- # # -------------------------------------------------------------------- #
import os import os
import logging import logging
import CsHelper from . import CsHelper
from CsFile import CsFile from .CsFile import CsFile
from CsProcess import CsProcess from .CsProcess import CsProcess
from CsApp import CsPasswdSvc from .CsApp import CsPasswdSvc
from CsAddress import CsDevice from .CsAddress import CsDevice
from CsRoute import CsRoute from .CsRoute import CsRoute
from CsStaticRoutes import CsStaticRoutes from .CsStaticRoutes import CsStaticRoutes
import socket import socket
from time import sleep from time import sleep
@ -435,7 +435,7 @@ class CsRedundant(object):
- public IPv6 for primary VR public NIC as its IPv6 gets lost on link down - public IPv6 for primary VR public NIC as its IPv6 gets lost on link down
""" """
dev = '' dev = ''
if dev == interface.get_device() or not ipv6 : if dev == interface.get_device() or not ipv6:
return return
dev = interface.get_device() dev = interface.get_device()
command = "ip -6 address show %s | grep 'inet6 %s'" % (dev, ipv6) command = "ip -6 address show %s | grep 'inet6 %s'" % (dev, ipv6)
@ -458,7 +458,7 @@ class CsRedundant(object):
- guest IPv6 gateway for primary VR guest NIC - guest IPv6 gateway for primary VR guest NIC
""" """
dev = '' dev = ''
if dev == interface.get_device() or not ipv6 : if dev == interface.get_device() or not ipv6:
return return
dev = interface.get_device() dev = interface.get_device()
command = "ip -6 address show %s | grep 'inet6 %s'" % (dev, ipv6) command = "ip -6 address show %s | grep 'inet6 %s'" % (dev, ipv6)
@ -495,7 +495,6 @@ class CsRedundant(object):
CsHelper.service("radvd", "disable") CsHelper.service("radvd", "disable")
logging.info(CsHelper.execute("systemctl status radvd")) logging.info(CsHelper.execute("systemctl status radvd"))
def _add_ipv6_guest_gateway(self): def _add_ipv6_guest_gateway(self):
""" """
Configure guest network gateway as IPv6 address for guest interface Configure guest network gateway as IPv6 address for guest interface

2
systemvm/debian/opt/cloud/bin/cs/CsRoute.py
View File

@ -15,7 +15,7 @@
# KIND, either express or implied. See the License for the # KIND, either express or implied. See the License for the
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
import CsHelper from . import CsHelper
import logging import logging

2
systemvm/debian/opt/cloud/bin/cs/CsRule.py
View File

@ -15,7 +15,7 @@
# KIND, either express or implied. See the License for the # KIND, either express or implied. See the License for the
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
import CsHelper from . import CsHelper
import logging import logging

4
systemvm/debian/opt/cloud/bin/cs/CsStaticRoutes.py
View File

@ -18,8 +18,8 @@
# under the License. # under the License.
import logging import logging
import CsHelper from . import CsHelper
from CsDatabag import CsDataBag from .CsDatabag import CsDataBag
class CsStaticRoutes(CsDataBag): class CsStaticRoutes(CsDataBag):

17
systemvm/debian/opt/cloud/bin/cs/CsVpcGuestNetwork.py
View File

@ -17,14 +17,15 @@
import logging import logging
import os.path import os.path
from cs.CsDatabag import CsDataBag from cs.CsDatabag import CsDataBag
from CsFile import CsFile from .CsFile import CsFile
import CsHelper from . import CsHelper
VPC_PUBLIC_INTERFACE = "eth1" VPC_PUBLIC_INTERFACE = "eth1"
RADVD_CONF = "/etc/radvd.conf" RADVD_CONF = "/etc/radvd.conf"
RADVD_CONF_NEW = "/etc/radvd.conf.new" RADVD_CONF_NEW = "/etc/radvd.conf.new"
class CsVpcGuestNetwork(CsDataBag): class CsVpcGuestNetwork(CsDataBag):
""" Manage Vpc Guest Networks """ """ Manage Vpc Guest Networks """
@ -53,13 +54,13 @@ class CsVpcGuestNetwork(CsDataBag):
CsHelper.execute("sysctl net.ipv6.conf." + device + ".use_tempaddr=0") CsHelper.execute("sysctl net.ipv6.conf." + device + ".use_tempaddr=0")
def add_address_route(self, entry): def add_address_route(self, entry):
if 'router_guest_ip6' in entry.keys() and entry['router_guest_ip6']: if 'router_guest_ip6' in list(entry.keys()) and entry['router_guest_ip6']:
self.enable_ipv6(entry['device']) self.enable_ipv6(entry['device'])
cidr_size = entry['router_guest_ip6_cidr'].split("/")[-1] cidr_size = entry['router_guest_ip6_cidr'].split("/")[-1]
full_addr = entry['router_guest_ip6_gateway'] + "/" + cidr_size full_addr = entry['router_guest_ip6_gateway'] + "/" + cidr_size
if not CsHelper.execute("ip -6 addr show dev %s | grep -w %s" % (entry['device'], full_addr)): if not CsHelper.execute("ip -6 addr show dev %s | grep -w %s" % (entry['device'], full_addr)):
CsHelper.execute("ip -6 addr add %s dev %s" % (full_addr, entry['device'])) CsHelper.execute("ip -6 addr add %s dev %s" % (full_addr, entry['device']))
if 'router_ip6' in entry.keys() and entry['router_ip6']: if 'router_ip6' in list(entry.keys()) and entry['router_ip6']:
self.__disable_dad(VPC_PUBLIC_INTERFACE) self.__disable_dad(VPC_PUBLIC_INTERFACE)
full_public_addr = entry['router_ip6'] + "/" + cidr_size full_public_addr = entry['router_ip6'] + "/" + cidr_size
if not CsHelper.execute("ip -6 addr show dev %s | grep -w %s" % (VPC_PUBLIC_INTERFACE, full_public_addr)): if not CsHelper.execute("ip -6 addr show dev %s | grep -w %s" % (VPC_PUBLIC_INTERFACE, full_public_addr)):
@ -70,11 +71,11 @@ class CsVpcGuestNetwork(CsDataBag):
return return
def remove_address_route(self, entry): def remove_address_route(self, entry):
if 'router_guest_ip6' in entry.keys() and entry['router_guest_ip6']: if 'router_guest_ip6' in list(entry.keys()) and entry['router_guest_ip6']:
cidr_size = entry['router_guest_ip6_cidr'].split("/")[-1] cidr_size = entry['router_guest_ip6_cidr'].split("/")[-1]
full_addr = entry['router_guest_ip6_gateway'] + "/" + cidr_size full_addr = entry['router_guest_ip6_gateway'] + "/" + cidr_size
CsHelper.execute("ip -6 addr del %s dev %s" % (full_addr, entry['device'])) CsHelper.execute("ip -6 addr del %s dev %s" % (full_addr, entry['device']))
if 'router_ip6' in entry.keys() and entry['router_ip6']: if 'router_ip6' in list(entry.keys()) and entry['router_ip6']:
full_public_addr = entry['router_ip6'] + "/" + cidr_size full_public_addr = entry['router_ip6'] + "/" + cidr_size
CsHelper.execute("ip -6 addr del %s dev %s" % (full_public_addr, VPC_PUBLIC_INTERFACE)) CsHelper.execute("ip -6 addr del %s dev %s" % (full_public_addr, VPC_PUBLIC_INTERFACE))
else: else:
@ -94,7 +95,7 @@ class CsVpcGuestNetwork(CsDataBag):
self.__disable_dad(device) self.__disable_dad(device)
def add_radvd_conf(self, entry): def add_radvd_conf(self, entry):
if 'router_guest_ip6' in entry.keys() and entry['router_guest_ip6']: if 'router_guest_ip6' in list(entry.keys()) and entry['router_guest_ip6']:
cidr_size = entry['router_guest_ip6_cidr'].split("/")[-1] cidr_size = entry['router_guest_ip6_cidr'].split("/")[-1]
full_addr = entry['router_guest_ip6_gateway'] + "/" + cidr_size full_addr = entry['router_guest_ip6_gateway'] + "/" + cidr_size
self.conf.append("interface %s" % entry['device']) self.conf.append("interface %s" % entry['device'])
@ -107,7 +108,7 @@ class CsVpcGuestNetwork(CsDataBag):
self.conf.append(" AdvOnLink on;") self.conf.append(" AdvOnLink on;")
self.conf.append(" AdvAutonomous on;") self.conf.append(" AdvAutonomous on;")
self.conf.append(" };") self.conf.append(" };")
if 'dns6' in entry.keys() and entry['dns6']: if 'dns6' in list(entry.keys()) and entry['dns6']:
for dns in entry['dns6'].split(","): for dns in entry['dns6'].split(","):
self.conf.append(" RDNSS %s" % dns) self.conf.append(" RDNSS %s" % dns)
self.conf.append(" {") self.conf.append(" {")

6
systemvm/debian/opt/cloud/bin/cs_dhcp.py
View File

@ -24,16 +24,16 @@ def merge(dbag, data):
# This seems desirable .... # This seems desirable ....
if "add" in data and data['add'] is False and "ipv4_address" in data: if "add" in data and data['add'] is False and "ipv4_address" in data:
if data['ipv4_address'] in dbag: if data['ipv4_address'] in dbag:
del(dbag[data['ipv4_address']]) del dbag[data['ipv4_address']]
else: else:
remove_keys = set() remove_keys = set()
for key, entry in dbag.iteritems(): for key, entry in dbag.items():
if key != 'id' and entry['mac_address'] == data['mac_address']: if key != 'id' and entry['mac_address'] == data['mac_address']:
remove_keys.add(key) remove_keys.add(key)
break break
for remove_key in remove_keys: for remove_key in remove_keys:
del(dbag[remove_key]) del dbag[remove_key]
dbag[data['ipv4_address']] = data dbag[data['ipv4_address']] = data

6
systemvm/debian/opt/cloud/bin/cs_firewallrules.py
View File

@ -25,8 +25,8 @@ def merge(dbag, data):
for rule in data['rules']: for rule in data['rules']:
id = str(rule['id']) id = str(rule['id'])
if rule['revoked']: if rule['revoked']:
if id in dbagc.keys(): if id in list(dbagc.keys()):
del(dbagc[id]) del dbagc[id]
elif id not in dbagc.keys(): elif id not in list(dbagc.keys()):
dbagc[id] = rule dbagc[id] = rule
return dbagc return dbagc

8
systemvm/debian/opt/cloud/bin/cs_forwardingrules.py
View File

@ -39,7 +39,7 @@ def merge(dbag, rules):
dbag[source_ip] = [newrule] dbag[source_ip] = [newrule]
elif rules["type"] == "forwardrules": elif rules["type"] == "forwardrules":
index = -1 index = -1
if source_ip in dbag.keys(): if source_ip in list(dbag.keys()):
for forward in dbag[source_ip]: for forward in dbag[source_ip]:
if ruleCompare(forward, newrule): if ruleCompare(forward, newrule):
index = dbag[source_ip].index(forward) index = dbag[source_ip].index(forward)
@ -51,15 +51,15 @@ def merge(dbag, rules):
dbag[source_ip] = [newrule] dbag[source_ip] = [newrule]
else: else:
if rules["type"] == "staticnatrules": if rules["type"] == "staticnatrules":
if source_ip in dbag.keys(): if source_ip in list(dbag.keys()):
del dbag[source_ip] del dbag[source_ip]
elif rules["type"] == "forwardrules": elif rules["type"] == "forwardrules":
if source_ip in dbag.keys(): if source_ip in list(dbag.keys()):
index = -1 index = -1
for forward in dbag[source_ip]: for forward in dbag[source_ip]:
if ruleCompare(forward, newrule): if ruleCompare(forward, newrule):
index = dbag[source_ip].index(forward) index = dbag[source_ip].index(forward)
print "removing index %s" % str(index) print("removing index %s" % str(index))
if not index == -1: if not index == -1:
del dbag[source_ip][index] del dbag[source_ip][index]

8
systemvm/debian/opt/cloud/bin/cs_guestnetwork.py
View File

@ -28,11 +28,11 @@ def merge(dbag, gn):
device_to_die = dbag[device][0] device_to_die = dbag[device][0]
try: try:
dbag[device].remove(device_to_die) dbag[device].remove(device_to_die)
except ValueError, e: except ValueError as e:
print "[WARN] cs_guestnetwork.py :: Error occurred removing item from databag. => %s" % device_to_die print("[WARN] cs_guestnetwork.py :: Error occurred removing item from databag. => %s" % device_to_die)
del(dbag[device]) del dbag[device]
else: else:
del(dbag[device]) del dbag[device]
else: else:
dbag.setdefault(device, []).append(gn) dbag.setdefault(device, []).append(gn)

2
systemvm/debian/opt/cloud/bin/cs_ip.py
View File

@ -57,7 +57,7 @@ def merge(dbag, ip):
ip['network'] = str(ipo.network) + '/' + str(ipo.prefixlen) ip['network'] = str(ipo.network) + '/' + str(ipo.prefixlen)
if 'mtu' in ip: if 'mtu' in ip:
ip['mtu'] = str(ip['mtu']) ip['mtu'] = str(ip['mtu'])
if 'nw_type' not in ip.keys(): if 'nw_type' not in list(ip.keys()):
ip['nw_type'] = 'public' ip['nw_type'] = 'public'
else: else:
ip['nw_type'] = ip['nw_type'].lower() ip['nw_type'] = ip['nw_type'].lower()

4
systemvm/debian/opt/cloud/bin/cs_remoteaccessvpn.py
View File

@ -20,8 +20,8 @@
def merge(dbag, vpn): def merge(dbag, vpn):
key = vpn['vpn_server_ip'] key = vpn['vpn_server_ip']
op = vpn['create'] op = vpn['create']
if key in dbag.keys() and not op: if key in list(dbag.keys()) and not op:
del(dbag[key]) del dbag[key]
else: else:
dbag[key] = vpn dbag[key] = vpn
return dbag return dbag

4
systemvm/debian/opt/cloud/bin/cs_site2sitevpn.py
View File

@ -20,8 +20,8 @@
def merge(dbag, vpn): def merge(dbag, vpn):
key = vpn['peer_gateway_ip'] key = vpn['peer_gateway_ip']
op = vpn['create'] op = vpn['create']
if key in dbag.keys() and not op: if key in list(dbag.keys()) and not op:
del(dbag[key]) del dbag[key]
else: else:
dbag[key] = vpn dbag[key] = vpn
return dbag return dbag

12
systemvm/debian/opt/cloud/bin/cs_vpnusers.py
View File

@ -22,26 +22,26 @@ import copy
def merge(dbag, data): def merge(dbag, data):
dbagc = copy.deepcopy(dbag) dbagc = copy.deepcopy(dbag)
print dbag print(dbag)
print data print(data)
if "vpn_users" not in data: if "vpn_users" not in data:
return dbagc return dbagc
# remove previously deleted user from the dict # remove previously deleted user from the dict
for user in dbagc.keys(): for user in list(dbagc.keys()):
if user == 'id': if user == 'id':
continue continue
userrec = dbagc[user] userrec = dbagc[user]
add = userrec['add'] add = userrec['add']
if not add: if not add:
del(dbagc[user]) del dbagc[user]
for user in data['vpn_users']: for user in data['vpn_users']:
username = user['user'] username = user['user']
add = user['add'] add = user['add']
if username not in dbagc.keys(): if username not in list(dbagc.keys()):
dbagc[username] = user dbagc[username] = user
elif username in dbagc.keys() and not add: elif username in list(dbagc.keys()) and not add:
dbagc[username] = user dbagc[username] = user
return dbagc return dbagc

4
systemvm/debian/opt/cloud/bin/diagnostics.py
View File

@ -34,8 +34,8 @@ def run_cmd(command):
return_code = 1 return_code = 1
finally: finally:
print('%s&&' % stdout.strip()) print('%s&&' % stdout.decode().strip())
print('%s&&' % stderr.strip()) print('%s&&' % stderr.decode().strip())
print('%s' % return_code) print('%s' % return_code)

6
systemvm/debian/opt/cloud/bin/filesystem_writable_check.py
View File

@ -28,17 +28,17 @@ def check_filesystem():
readOnly1 = bool(stat1.f_flag & ST_RDONLY) readOnly1 = bool(stat1.f_flag & ST_RDONLY)
if (readOnly1): if (readOnly1):
print "Read-only file system : monitor results (/root) file system is mounted as read-only" print("Read-only file system : monitor results (/root) file system is mounted as read-only")
exit(1) exit(1)
stat2 = os.statvfs('/var/cache/cloud') stat2 = os.statvfs('/var/cache/cloud')
readOnly2 = bool(stat2.f_flag & ST_RDONLY) readOnly2 = bool(stat2.f_flag & ST_RDONLY)
if (readOnly2): if (readOnly2):
print "Read-only file system : config info (/var/cache/cloud) file system is mounted as read-only" print("Read-only file system : config info (/var/cache/cloud) file system is mounted as read-only")
exit(1) exit(1)
print "file system is writable" print("file system is writable")
exit(0) exit(0)

8
systemvm/debian/opt/cloud/bin/get_diagnostics_files.py
View File

@ -65,7 +65,7 @@ def zip_files(files):
cleanup(files_from_shell_commands) cleanup(files_from_shell_commands)
generate_retrieved_files_txt(zf, files_found_list, files_not_found_list) generate_retrieved_files_txt(zf, files_found_list, files_not_found_list)
zf.close() zf.close()
print zf_name print(zf_name)
def get_cmd(script): def get_cmd(script):
@ -102,7 +102,7 @@ def execute_shell_script(script):
p = sp.Popen(cmd, shell=True, stdout=sp.PIPE, stderr=sp.PIPE) p = sp.Popen(cmd, shell=True, stdout=sp.PIPE, stderr=sp.PIPE)
stdout, stderr = p.communicate() stdout, stderr = p.communicate()
return_code = p.returncode return_code = p.returncode
if return_code is 0: if return_code == 0:
f.write(stdout) f.write(stdout)
else: else:
f.write(stderr) f.write(stderr)
@ -129,9 +129,9 @@ def generate_retrieved_files_txt(zip_file, files_found, files_not_found):
try: try:
with open(output_file, 'wb', 0) as man: with open(output_file, 'wb', 0) as man:
for i in files_found: for i in files_found:
man.write(i + '\n') man.write((i + '\n').encode())
for j in files_not_found: for j in files_not_found:
man.write(j + 'File Not Found!!\n') man.write((j + ' File Not Found!!\n').encode())
zip_file.write(output_file, output_file) zip_file.write(output_file, output_file)
finally: finally:
cleanup_cmd = "rm -f %s" % output_file cleanup_cmd = "rm -f %s" % output_file

8
systemvm/debian/opt/cloud/bin/merge.py
View File

@ -158,7 +158,7 @@ class updateDataBag:
dp['mtu'] = str(d['mtu']) dp['mtu'] = str(d['mtu'])
qf = QueueFile() qf = QueueFile()
qf.load({'ip_address': [dp], 'type': 'ips'}) qf.load({'ip_address': [dp], 'type': 'ips'})
if 'domain_name' not in d.keys() or d['domain_name'] == '': if 'domain_name' not in list(d.keys()) or d['domain_name'] == '':
d['domain_name'] = "cloudnine.internal" d['domain_name'] = "cloudnine.internal"
return cs_guestnetwork.merge(dbag, d) return cs_guestnetwork.merge(dbag, d)
@ -227,7 +227,7 @@ class updateDataBag:
def processCLItem(self, num, nw_type): def processCLItem(self, num, nw_type):
key = 'eth' + num + 'ip' key = 'eth' + num + 'ip'
dp = {} dp = {}
if(key in self.qFile.data['cmd_line']): if key in self.qFile.data['cmd_line']:
dp['public_ip'] = self.qFile.data['cmd_line'][key] dp['public_ip'] = self.qFile.data['cmd_line'][key]
dp['netmask'] = self.qFile.data['cmd_line']['eth' + num + 'mask'] dp['netmask'] = self.qFile.data['cmd_line']['eth' + num + 'mask']
dp['source_nat'] = False dp['source_nat'] = False
@ -236,7 +236,7 @@ class updateDataBag:
if nw_type == "public": if nw_type == "public":
dp['gateway'] = self.qFile.data['cmd_line']['gateway'] dp['gateway'] = self.qFile.data['cmd_line']['gateway']
else: else:
if('localgw' in self.qFile.data['cmd_line']): if 'localgw' in self.qFile.data['cmd_line']:
dp['gateway'] = self.qFile.data['cmd_line']['localgw'] dp['gateway'] = self.qFile.data['cmd_line']['localgw']
else: else:
dp['gateway'] = '' dp['gateway'] = ''
@ -252,7 +252,7 @@ class updateDataBag:
def process_ipaliases(self, dbag): def process_ipaliases(self, dbag):
nic_dev = None nic_dev = None
# Should be a way to deal with this better # Should be a way to deal with this better
for intf, data in dbag.items(): for intf, data in list(dbag.items()):
if intf == 'id': if intf == 'id':
continue continue
elif any([net['nw_type'] == 'guest' for net in data]): elif any([net['nw_type'] == 'guest' for net in data]):

18
systemvm/debian/opt/cloud/bin/passwd_server_ip.py
View File

@ -31,10 +31,10 @@ import os
import sys import sys
import syslog import syslog
import threading import threading
import urlparse import urllib.parse
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer from http.server import BaseHTTPRequestHandler, HTTPServer
from SocketServer import ThreadingMixIn #, ForkingMixIn from socketserver import ThreadingMixIn #, ForkingMixIn
passMap = {} passMap = {}
@ -55,7 +55,7 @@ def initToken():
with open(getTokenFile(), 'r') as f: with open(getTokenFile(), 'r') as f:
secureToken = f.read() secureToken = f.read()
if not secureToken: if not secureToken:
secureToken = binascii.hexlify(os.urandom(16)) secureToken = binascii.hexlify(os.urandom(16)).decode()
with open(getTokenFile(), 'w') as f: with open(getTokenFile(), 'w') as f:
f.write(secureToken) f.write(secureToken)
@ -64,7 +64,7 @@ def checkToken(token):
def loadPasswordFile(): def loadPasswordFile():
try: try:
with file(getPasswordFile()) as f: with open(getPasswordFile()) as f:
for line in f: for line in f:
if '=' not in line: continue if '=' not in line: continue
key, value = line.strip().split('=', 1) key, value = line.strip().split('=', 1)
@ -75,11 +75,11 @@ def loadPasswordFile():
def savePasswordFile(): def savePasswordFile():
with lock: with lock:
try: try:
with file(getPasswordFile(), 'w') as f: with open(getPasswordFile(), 'w') as f:
for ip in passMap: for ip in passMap:
f.write('%s=%s\n' % (ip, passMap[ip])) f.write('%s=%s\n' % (ip, passMap[ip]))
f.close() f.close()
except IOError, e: except IOError as e:
syslog.syslog('serve_password: Unable to save to password file %s' % e) syslog.syslog('serve_password: Unable to save to password file %s' % e)
def getPassword(ip): def getPassword(ip):
@ -117,7 +117,7 @@ class PasswordRequestHandler(BaseHTTPRequestHandler):
self.wfile.write('saved_password') self.wfile.write('saved_password')
syslog.syslog('serve_password: requested password not found for %s' % clientAddress) syslog.syslog('serve_password: requested password not found for %s' % clientAddress)
else: else:
self.wfile.write(password) self.wfile.write(password.encode())
syslog.syslog('serve_password: password sent to %s' % clientAddress) syslog.syslog('serve_password: password sent to %s' % clientAddress)
elif requestType == 'saved_password': elif requestType == 'saved_password':
removePassword(clientAddress) removePassword(clientAddress)
@ -192,7 +192,7 @@ def serve(HandlerClass = PasswordRequestHandler,
except KeyboardInterrupt: except KeyboardInterrupt:
syslog.syslog('serve_password shutting down') syslog.syslog('serve_password shutting down')
passwordServer.socket.close() passwordServer.socket.close()
except Exception, e: except Exception as e:
syslog.syslog('serve_password hit exception %s -- died' % e) syslog.syslog('serve_password hit exception %s -- died' % e)
passwordServer.socket.close() passwordServer.socket.close()

1
systemvm/debian/opt/cloud/bin/setup/cksnode.sh
View File

@ -72,3 +72,4 @@ setup_k8s_node() {
} }
setup_k8s_node setup_k8s_node
. /opt/cloud/bin/setup/patch.sh && patch_sshd_config

2
systemvm/debian/opt/cloud/bin/setup/consoleproxy.sh
View File

@ -45,3 +45,5 @@ setup_console_proxy() {
} }
setup_console_proxy setup_console_proxy
# System VMs are patched during bootstrap
. /opt/cloud/bin/setup/patch.sh && patch_system_vm

1
systemvm/debian/opt/cloud/bin/setup/dhcpsrvr.sh
View File

@ -52,3 +52,4 @@ then
exit 1 exit 1
fi fi
setup_dhcpsrvr setup_dhcpsrvr
. /opt/cloud/bin/setup/patch.sh && patch_router

1
systemvm/debian/opt/cloud/bin/setup/elbvm.sh
View File

@ -41,3 +41,4 @@ then
exit 1 exit 1
fi fi
setup_elbvm setup_elbvm
. /opt/cloud/bin/setup/patch.sh && patch_router

1
systemvm/debian/opt/cloud/bin/setup/ilbvm.sh
View File

@ -44,3 +44,4 @@ then
exit 1 exit 1
fi fi
setup_ilbvm setup_ilbvm
. /opt/cloud/bin/setup/patch.sh && patch_router

128
systemvm/debian/opt/cloud/bin/setup/patch.sh Executable file
View File

@ -0,0 +1,128 @@
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
log_it() {
echo "$(date) $@" >> /var/log/cloud.log
}
patch_sshd_config() {
if `! ssh -Q PubkeyAcceptedAlgorithms >/dev/null 2>&1` && `grep ^PubkeyAcceptedAlgorithms /etc/ssh/sshd_config >/dev/null`; then
# "PubkeyAcceptedAlgorithms=+ssh-rsa" is added to /etc/ssh/sshd_config in 4.20.0 systemvm template
# However, it is not supported in old systemvm templates
# If the system vm is created from an old systemvm template, remove it from /etc/ssh/sshd_config
# No need to restart ssh if it is running well
log_it "Removing PubkeyAcceptedAlgorithms=+ssh-rsa from /etc/ssh/sshd_config as it is not supported"
sed -i "/PubkeyAcceptedAlgorithms=+ssh-rsa/d" /etc/ssh/sshd_config
if ! systemctl is-active ssh > /dev/null; then
systemctl restart ssh
fi
elif `ssh -Q PubkeyAcceptedAlgorithms >/dev/null 2>&1` && `! grep ^PubkeyAcceptedAlgorithms /etc/ssh/sshd_config >/dev/null`; then
log_it "Adding PubkeyAcceptedAlgorithms=+ssh-rsa to sshd_config"
sed -i "/PubkeyAuthentication yes/aPubkeyAcceptedAlgorithms=+ssh-rsa" /etc/ssh/sshd_config
systemctl restart ssh
fi
}
patch_router() {
local patchfile="/var/cache/cloud/agent.zip"
local logfile="/var/log/patchrouter.log"
rm /usr/local/cloud/systemvm -rf
mkdir -p /usr/local/cloud/systemvm
ls -lrt $patchfile
log_it "Unziping $patchfile"
echo "All" | unzip $patchfile -d /usr/local/cloud/systemvm >>$logfile 2>&1
find /usr/local/cloud/systemvm/ -name \*.sh | xargs chmod 555
patch_sshd_config
install_packages
}
patch_system_vm() {
patch_sshd_config
install_packages
}
install_packages() {
PACKAGES_FOLDER="/usr/local/cloud/systemvm/packages"
PACKAGES_INI="$PACKAGES_FOLDER/packages.ini"
declare -A package_properties
if [ -d $PACKAGES_FOLDER ] && [ -f $PACKAGES_INI ]; then
while read -r line; do
if [[ "$line" =~ ^(\[)(.*)(\])$ ]]; then
install_package
package_properties=
else
key=$(echo $line | cut -d '=' -f1)
value=$(echo $line | cut -d '=' -f2)
if [ "$key" != "" ]; then
package_properties[$key]=$value
fi
fi
done <$PACKAGES_INI
fi
export DEBIAN_FRONTEND=noninteractive
install_package
}
install_package() {
local os=${package_properties["debian_os"]}
if [ "$os" == "" ]; then
return
fi
local DEBIAN_RELEASE=$(lsb_release -rs)
if [ "$os" != "$DEBIAN_RELEASE" ]; then
log_it "Skipped the installation of package $package on Debian $DEBIAN_RELEASE as it can only be installed on Debian $os."
return
fi
local package=${package_properties["package_name"]}
local file=${package_properties["file_name"]}
if [ -z "$package" ] || [ -z "$file" ]; then
log_it "Skipped the installation due to empty package of file name (package name: $package, file name: $file)."
return
fi
dpkg-query -s $package >/dev/null 2>&1
if [ $? -eq 0 ]; then
log_it "Skipped the installation as package $package has already been installed."
return
fi
local conflicts=${package_properties["conflicted_packages"]}
if [ "$conflicts" != "" ]; then
log_it "Removing conflicted packages \"$conflicts\" before installing package $package"
apt remove -y "$conflicts"
if [ $? -eq 0 ]; then
log_it "Removed conflicted package(s) \"$conflicts\" before installing package $package"
else
log_it "Failed to remove conflicted package(s) \"$conflicts\" before installing package $package"
fi
fi
PACKAGES_FOLDER="/usr/local/cloud/systemvm/packages"
log_it "Installing package $package from file $PACKAGES_FOLDER/$file"
dpkg -i $PACKAGES_FOLDER/$file
if [ $? -eq 0 ]; then
log_it "Installed package $package from file $PACKAGES_FOLDER/$file"
else
log_it "Failed to install package $package from file $PACKAGES_FOLDER/$file"
fi
}

1
systemvm/debian/opt/cloud/bin/setup/router.sh
View File

@ -101,3 +101,4 @@ then
exit 1 exit 1
fi fi
setup_router setup_router
. /opt/cloud/bin/setup/patch.sh && patch_router

2
systemvm/debian/opt/cloud/bin/setup/secstorage.sh
View File

@ -87,3 +87,5 @@ HTTP
} }
setup_secstorage setup_secstorage
# System VMs are patched during bootstrap
. /opt/cloud/bin/setup/patch.sh && patch_system_vm

1
systemvm/debian/opt/cloud/bin/setup/vpcrouter.sh
View File

@ -129,3 +129,4 @@ then
exit 1 exit 1
fi fi
setup_vpcrouter setup_vpcrouter
. /opt/cloud/bin/setup/patch.sh && patch_router

2
systemvm/debian/opt/cloud/bin/update_config.py
View File

@ -62,7 +62,7 @@ def is_guestnet_configured(guestnet_dict, keys):
existing_keys = [] existing_keys = []
new_eth_key = None new_eth_key = None
for k1, v1 in guestnet_dict.iteritems(): for k1, v1 in guestnet_dict.items():
if k1 in keys and len(v1) > 0: if k1 in keys and len(v1) > 0:
existing_keys.append(k1) existing_keys.append(k1)

25
systemvm/debian/opt/cloud/bin/vmdata.py
View File

@ -31,7 +31,7 @@ def main(argv):
try: try:
opts, args = getopt.getopt(argv, "f:d:") opts, args = getopt.getopt(argv, "f:d:")
except getopt.GetoptError: except getopt.GetoptError:
print 'params: -f <filename> -d <b64jsondata>' print('params: -f <filename> -d <b64jsondata>')
sys.exit(2) sys.exit(2)
for opt, arg in opts: for opt, arg in opts:
if opt == '-f': if opt == '-f':
@ -46,7 +46,7 @@ def main(argv):
elif b64data != '': elif b64data != '':
json_data = json.loads(base64.b64decode(b64data)) json_data = json.loads(base64.b64decode(b64data))
else: else:
print '-f <filename> or -d <b64jsondata> required' print('-f <filename> or -d <b64jsondata> required')
sys.exit(2) sys.exit(2)
for ip in json_data: for ip in json_data:
@ -94,20 +94,23 @@ def createfile(ip, folder, file, data):
fh = open(dest, "w") fh = open(dest, "w")
exflock(fh) exflock(fh)
if data is not None: if data is not None:
fh.write(data) if isinstance(data, str):
fh.write(data)
elif isinstance(data, bytes):
fh.write(data.decode())
else: else:
fh.write("") fh.write("")
unflock(fh) unflock(fh)
fh.close() fh.close()
os.chmod(dest, 0644) os.chmod(dest, 0o644)
if folder == "metadata" or folder == "meta-data": if folder == "metadata" or folder == "meta-data":
try: try:
os.makedirs(metamanifestdir, 0755) os.makedirs(metamanifestdir, 0o755)
except OSError as e: except OSError as e:
# error 17 is already exists, we do it this way for concurrency # error 17 is already exists, we do it this way for concurrency
if e.errno != 17: if e.errno != 17:
print "failed to make directories " + metamanifestdir + " due to :" + e.strerror print("failed to make directories " + metamanifestdir + " due to :" + e.strerror)
sys.exit(1) sys.exit(1)
if os.path.exists(metamanifest): if os.path.exists(metamanifest):
fh = open(metamanifest, "r+a") fh = open(metamanifest, "r+a")
@ -124,7 +127,7 @@ def createfile(ip, folder, file, data):
fh.close() fh.close()
if os.path.exists(metamanifest): if os.path.exists(metamanifest):
os.chmod(metamanifest, 0644) os.chmod(metamanifest, 0o644)
def htaccess(ip, folder, file): def htaccess(ip, folder, file):
@ -133,11 +136,11 @@ def htaccess(ip, folder, file):
htaccessFile = htaccessFolder+"/.htaccess" htaccessFile = htaccessFolder+"/.htaccess"
try: try:
os.makedirs(htaccessFolder, 0755) os.makedirs(htaccessFolder, 0o755)
except OSError as e: except OSError as e:
# error 17 is already exists, we do it this way for sake of concurrency # error 17 is already exists, we do it this way for sake of concurrency
if e.errno != 17: if e.errno != 17:
print "failed to make directories " + htaccessFolder + " due to :" + e.strerror print("failed to make directories " + htaccessFolder + " due to :" + e.strerror)
sys.exit(1) sys.exit(1)
fh = open(htaccessFile, "w") fh = open(htaccessFile, "w")
@ -151,7 +154,7 @@ def exflock(file):
try: try:
flock(file, LOCK_EX) flock(file, LOCK_EX)
except IOError as e: except IOError as e:
print "failed to lock file" + file.name + " due to : " + e.strerror print("failed to lock file" + file.name + " due to : " + e.strerror)
sys.exit(1) sys.exit(1)
return True return True
@ -160,7 +163,7 @@ def unflock(file):
try: try:
flock(file, LOCK_UN) flock(file, LOCK_UN)
except IOError as e: except IOError as e:
print "failed to unlock file" + file.name + " due to : " + e.strerror print("failed to unlock file" + file.name + " due to : " + e.strerror)
sys.exit(1) sys.exit(1)
return True return True

14
systemvm/debian/root/health_checks/cpu_usage_check.py
View File

@ -28,7 +28,7 @@ def main():
data = entries[0] data = entries[0]
if "maxCpuUsage" not in data: if "maxCpuUsage" not in data:
print "Missing maxCpuUsage in health_checks_data systemThresholds, skipping" print("Missing maxCpuUsage in health_checks_data systemThresholds, skipping")
exit(0) exit(0)
maxCpuUsage = float(data["maxCpuUsage"]) maxCpuUsage = float(data["maxCpuUsage"])
@ -38,16 +38,16 @@ def main():
"sub(\"%\", \"\", idle); printf \"%.2f\", 100 - idle }'" "sub(\"%\", \"\", idle); printf \"%.2f\", 100 - idle }'"
pout = Popen(cmd, shell=True, stdout=PIPE) pout = Popen(cmd, shell=True, stdout=PIPE)
if pout.wait() == 0: if pout.wait() == 0:
currentUsage = float(pout.communicate()[0].strip()) currentUsage = float(pout.communicate()[0].decode().strip())
if currentUsage > maxCpuUsage: if currentUsage > maxCpuUsage:
print "CPU Usage " + str(currentUsage) + \ print("CPU Usage " + str(currentUsage) +
"% has crossed threshold of " + str(maxCpuUsage) + "%" "% has crossed threshold of " + str(maxCpuUsage) + "%")
exit(1) exit(1)
print "CPU Usage within limits with current at " \ print("CPU Usage within limits with current at "
+ str(currentUsage) + "%" + str(currentUsage) + "%")
exit(0) exit(0)
else: else:
print "Failed to retrieve cpu usage using " + cmd print("Failed to retrieve cpu usage using " + cmd)
exit(1) exit(1)

6
systemvm/debian/root/health_checks/dhcp_check.py
View File

@ -24,7 +24,7 @@ def main():
vMs = getHealthChecksData("virtualMachines") vMs = getHealthChecksData("virtualMachines")
if vMs is None or len(vMs) == 0: if vMs is None or len(vMs) == 0:
print "No VMs running data available, skipping" print("No VMs running data available, skipping")
exit(0) exit(0)
try: try:
@ -64,10 +64,10 @@ def main():
failureMessage = failureMessage + entry + ", " failureMessage = failureMessage + entry + ", "
if failedCheck: if failedCheck:
print failureMessage[:-2] print(failureMessage[:-2])
exit(1) exit(1)
else: else:
print "All " + str(COUNT) + " VMs are present in dhcphosts.txt" print("All " + str(COUNT) + " VMs are present in dhcphosts.txt")
exit(0) exit(0)

6
systemvm/debian/root/health_checks/disk_space_check.py
View File

@ -27,7 +27,7 @@ def main():
data = entries[0] data = entries[0]
if "minDiskNeeded" not in data: if "minDiskNeeded" not in data:
print "Missing minDiskNeeded in health_checks_data systemThresholds, skipping" print("Missing minDiskNeeded in health_checks_data systemThresholds, skipping")
exit(0) exit(0)
minDiskNeeded = float(data["minDiskNeeded"]) * 1024 minDiskNeeded = float(data["minDiskNeeded"]) * 1024
@ -35,10 +35,10 @@ def main():
freeSpace = (s.f_bavail * s.f_frsize) / 1024 freeSpace = (s.f_bavail * s.f_frsize) / 1024
if (freeSpace < minDiskNeeded): if (freeSpace < minDiskNeeded):
print "Insufficient free space is " + str(freeSpace/1024) + " MB" print("Insufficient free space is " + str(freeSpace/1024) + " MB")
exit(1) exit(1)
else: else:
print "Sufficient free space is " + str(freeSpace/1024) + " MB" print("Sufficient free space is " + str(freeSpace/1024) + " MB")
exit(0) exit(0)

6
systemvm/debian/root/health_checks/dns_check.py
View File

@ -24,7 +24,7 @@ def main():
vMs = getHealthChecksData("virtualMachines") vMs = getHealthChecksData("virtualMachines")
if vMs is None or len(vMs) == 0: if vMs is None or len(vMs) == 0:
print "No VMs running data available, skipping" print("No VMs running data available, skipping")
exit(0) exit(0)
with open('/etc/hosts', 'r') as hostsFile: with open('/etc/hosts', 'r') as hostsFile:
@ -51,10 +51,10 @@ def main():
failureMessage = failureMessage + vM["ip"] + " " + vM["vmName"] + ", " failureMessage = failureMessage + vM["ip"] + " " + vM["vmName"] + ", "
if failedCheck: if failedCheck:
print failureMessage[:-2] print(failureMessage[:-2])
exit(1) exit(1)
else: else:
print "All " + str(COUNT) + " VMs are present in /etc/hosts" print("All " + str(COUNT) + " VMs are present in /etc/hosts")
exit(0) exit(0)

8
systemvm/debian/root/health_checks/gateways_check.py
View File

@ -24,7 +24,7 @@ from utility import getHealthChecksData
def main(): def main():
gws = getHealthChecksData("gateways") gws = getHealthChecksData("gateways")
if gws is None and len(gws) == 0: if gws is None and len(gws) == 0:
print "No gateways data available, skipping" print("No gateways data available, skipping")
exit(0) exit(0)
unreachableGateWays = [] unreachableGateWays = []
@ -44,11 +44,11 @@ def main():
unreachableGateWays.append(gw) unreachableGateWays.append(gw)
if len(unreachableGateWays) == 0: if len(unreachableGateWays) == 0:
print "All " + str(len(gwsList)) + " gateways are reachable via ping" print("All " + str(len(gwsList)) + " gateways are reachable via ping")
exit(0) exit(0)
else: else:
print "Unreachable gateways found-" print("Unreachable gateways found-")
print unreachableGateWays print(unreachableGateWays)
exit(1) exit(1)

22
systemvm/debian/root/health_checks/haproxy_check.py
View File

@ -23,7 +23,7 @@ from utility import getHealthChecksData, formatPort
def checkMaxconn(haproxyData, haCfgSections): def checkMaxconn(haproxyData, haCfgSections):
if "maxconn" in haproxyData and "maxconn" in haCfgSections["global"]: if "maxconn" in haproxyData and "maxconn" in haCfgSections["global"]:
if haproxyData["maxconn"] != haCfgSections["global"]["maxconn"][0].strip(): if haproxyData["maxconn"] != haCfgSections["global"]["maxconn"][0].strip():
print "global maxconn mismatch occurred" print("global maxconn mismatch occurred")
return False return False
return True return True
@ -38,26 +38,26 @@ def checkLoadBalance(haproxyData, haCfgSections):
secName = "listen " + srcServer secName = "listen " + srcServer
if secName not in haCfgSections: if secName not in haCfgSections:
print "Missing section for load balancing " + secName + "\n" print("Missing section for load balancing " + secName + "\n")
correct = False correct = False
else: else:
cfgSection = haCfgSections[secName] cfgSection = haCfgSections[secName]
if "server" in cfgSection: if "server" in cfgSection:
if lbSec["algorithm"] != cfgSection["balance"][0]: if lbSec["algorithm"] != cfgSection["balance"][0]:
print "Incorrect balance method for " + secName + \ print("Incorrect balance method for " + secName +
"Expected : " + lbSec["algorithm"] + \ "Expected : " + lbSec["algorithm"] +
" but found " + cfgSection["balance"][0] + "\n" " but found " + cfgSection["balance"][0] + "\n")
correct = False correct = False
bindStr = lbSec["sourceIp"] + ":" + formatPort(lbSec["sourcePortStart"], lbSec["sourcePortEnd"]) bindStr = lbSec["sourceIp"] + ":" + formatPort(lbSec["sourcePortStart"], lbSec["sourcePortEnd"])
if cfgSection["bind"][0] != bindStr: if cfgSection["bind"][0] != bindStr:
print "Incorrect bind string found. Expected " + bindStr + " but found " + cfgSection["bind"][0] + "." print("Incorrect bind string found. Expected " + bindStr + " but found " + cfgSection["bind"][0] + ".")
correct = False correct = False
if (lbSec["sourcePortStart"] == "80" and lbSec["sourcePortEnd"] == "80" and lbSec["keepAliveEnabled"] == "false") \ if (lbSec["sourcePortStart"] == "80" and lbSec["sourcePortEnd"] == "80" and lbSec["keepAliveEnabled"] == "false") \
or (lbSec["stickiness"].find("AppCookie") != -1 or lbSec["stickiness"].find("LbCookie") != -1): or (lbSec["stickiness"].find("AppCookie") != -1 or lbSec["stickiness"].find("LbCookie") != -1):
if not ("mode" in cfgSection and cfgSection["mode"][0] == "http"): if not ("mode" in cfgSection and cfgSection["mode"][0] == "http"):
print "Expected HTTP mode but not found" print("Expected HTTP mode but not found")
correct = False correct = False
expectedServerIps = lbSec["vmIps"].split(" ") expectedServerIps = lbSec["vmIps"].split(" ")
@ -74,7 +74,7 @@ def checkLoadBalance(haproxyData, haCfgSections):
if not foundPattern: if not foundPattern:
correct = False correct = False
print "Missing load balancing for " + pattern + ". " print("Missing load balancing for " + pattern + ". ")
return correct return correct
@ -86,7 +86,7 @@ def main():
''' '''
haproxyData = getHealthChecksData("haproxyData") haproxyData = getHealthChecksData("haproxyData")
if haproxyData is None or len(haproxyData) == 0: if haproxyData is None or len(haproxyData) == 0:
print "No data provided to check, skipping" print("No data provided to check, skipping")
exit(0) exit(0)
with open("/etc/haproxy/haproxy.cfg", 'r') as haCfgFile: with open("/etc/haproxy/haproxy.cfg", 'r') as haCfgFile:
@ -94,7 +94,7 @@ def main():
haCfgFile.close() haCfgFile.close()
if len(haCfgLines) == 0: if len(haCfgLines) == 0:
print "Unable to read config file /etc/haproxy/haproxy.cfg" print("Unable to read config file /etc/haproxy/haproxy.cfg")
exit(1) exit(1)
haCfgSections = {} haCfgSections = {}
@ -123,7 +123,7 @@ def main():
checkLbRules = checkLoadBalance(haproxyData, haCfgSections) checkLbRules = checkLoadBalance(haproxyData, haCfgSections)
if checkMaxConn and checkLbRules: if checkMaxConn and checkLbRules:
print "All checks pass" print("All checks pass")
exit(0) exit(0)
else: else:
exit(1) exit(1)

8
systemvm/debian/root/health_checks/iptables_check.py
View File

@ -24,7 +24,7 @@ from utility import getHealthChecksData, formatPort
def main(): def main():
portForwards = getHealthChecksData("portForwarding") portForwards = getHealthChecksData("portForwarding")
if portForwards is None or len(portForwards) == 0: if portForwards is None or len(portForwards) == 0:
print "No portforwarding rules provided to check, skipping" print("No portforwarding rules provided to check, skipping")
exit(0) exit(0)
failedCheck = False failedCheck = False
@ -47,7 +47,7 @@ def main():
"for fetching rules by " + fetchIpTableEntriesCmd + "\n" "for fetching rules by " + fetchIpTableEntriesCmd + "\n"
continue continue
ipTablesMatchingEntries = pout.communicate()[0].strip().split('\n') ipTablesMatchingEntries = pout.communicate()[0].decode().strip().split('\n')
for pfEntryListExpected in entriesExpected: for pfEntryListExpected in entriesExpected:
foundPfEntryList = False foundPfEntryList = False
for ipTableEntry in ipTablesMatchingEntries: for ipTableEntry in ipTablesMatchingEntries:
@ -68,10 +68,10 @@ def main():
failureMessage = failureMessage + str(pfEntryListExpected) + "\n" failureMessage = failureMessage + str(pfEntryListExpected) + "\n"
if failedCheck: if failedCheck:
print failureMessage print(failureMessage)
exit(1) exit(1)
else: else:
print "Found all entries (count " + str(len(portForwards)) + ") in iptables" print("Found all entries (count " + str(len(portForwards)) + ") in iptables")
exit(0) exit(0)

16
systemvm/debian/root/health_checks/memory_usage_check.py
View File

@ -28,8 +28,8 @@ def main():
data = entries[0] data = entries[0]
if "maxMemoryUsage" not in data: if "maxMemoryUsage" not in data:
print "Missing maxMemoryUsage in health_checks_data " + \ print("Missing maxMemoryUsage in health_checks_data " +
"systemThresholds, skipping" "systemThresholds, skipping")
exit(0) exit(0)
maxMemoryUsage = float(data["maxMemoryUsage"]) maxMemoryUsage = float(data["maxMemoryUsage"])
@ -37,16 +37,16 @@ def main():
pout = Popen(cmd, shell=True, stdout=PIPE) pout = Popen(cmd, shell=True, stdout=PIPE)
if pout.wait() == 0: if pout.wait() == 0:
currentUsage = float(pout.communicate()[0].strip()) currentUsage = float(pout.communicate()[0].decode().strip())
if currentUsage > maxMemoryUsage: if currentUsage > maxMemoryUsage:
print "Memory Usage " + str(currentUsage) + \ print("Memory Usage " + str(currentUsage) +
"% has crossed threshold of " + str(maxMemoryUsage) + "%" "% has crossed threshold of " + str(maxMemoryUsage) + "%")
exit(1) exit(1)
print "Memory Usage within limits with current at " + \ print("Memory Usage within limits with current at " +
str(currentUsage) + "%" str(currentUsage) + "%")
exit(0) exit(0)
else: else:
print "Failed to retrieve memory usage using " + cmd print("Failed to retrieve memory usage using " + cmd)
exit(1) exit(1)

20
systemvm/debian/root/health_checks/router_version_check.py
View File

@ -41,7 +41,7 @@ def main():
data = entries[0] data = entries[0]
if len(data) == 0: if len(data) == 0:
print "Missing routerVersion in health_checks_data, skipping" print("Missing routerVersion in health_checks_data, skipping")
exit(0) exit(0)
templateVersionMatches = True templateVersionMatches = True
@ -52,11 +52,11 @@ def main():
releaseFile = "/etc/cloudstack-release" releaseFile = "/etc/cloudstack-release"
found = getFirstLine(releaseFile) found = getFirstLine(releaseFile)
if found is None: if found is None:
print "Release version not yet setup at " + releaseFile +\ print("Release version not yet setup at " + releaseFile +
", skipping." ", skipping.")
elif expected != found: elif expected != found:
print "Template Version mismatch. Expected: " + \ print("Template Version mismatch. Expected: " +
expected + ", found: " + found expected + ", found: " + found)
templateVersionMatches = False templateVersionMatches = False
if "scriptsVersion" in data: if "scriptsVersion" in data:
@ -64,15 +64,15 @@ def main():
sigFile = "/var/cache/cloud/cloud-scripts-signature" sigFile = "/var/cache/cloud/cloud-scripts-signature"
found = getFirstLine(sigFile) found = getFirstLine(sigFile)
if found is None: if found is None:
print "Scripts signature is not yet setup at " + sigFile +\ print("Scripts signature is not yet setup at " + sigFile +
", skipping" ", skipping")
if expected != found: if expected != found:
print "Scripts Version mismatch. Expected: " + \ print("Scripts Version mismatch. Expected: " +
expected + ", found: " + found expected + ", found: " + found)
scriptVersionMatches = False scriptVersionMatches = False
if templateVersionMatches and scriptVersionMatches: if templateVersionMatches and scriptVersionMatches:
print "Template and scripts version match successful" print("Template and scripts version match successful")
exit(0) exit(0)
else: else:
exit(1) exit(1)

2
systemvm/debian/root/health_checks/utility/__init__.py
View File

@ -16,4 +16,4 @@
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
from sharedFunctions import getHealthChecksData, formatPort from .sharedFunctions import getHealthChecksData, formatPort

16
systemvm/debian/root/monitorServices.py
View File

@ -16,7 +16,7 @@
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
from ConfigParser import SafeConfigParser from configparser import ConfigParser
from subprocess import * from subprocess import *
from datetime import datetime from datetime import datetime
import time import time
@ -56,7 +56,7 @@ def getServicesConfig( config_file_path = "/etc/monitor.conf" ):
""" """
process_dict = {} process_dict = {}
parser = SafeConfigParser() parser = ConfigParser()
parser.read( config_file_path ) parser.read( config_file_path )
@ -81,7 +81,7 @@ def printd (msg):
f.seek(0, 2) f.seek(0, 2)
f.write(str(msg)+"\n") f.write(str(msg)+"\n")
f.close() f.close()
print str(msg) print(str(msg))
def raisealert(severity, msg, process_name=None): def raisealert(severity, msg, process_name=None):
""" Writes the alert message""" """ Writes the alert message"""
@ -96,7 +96,7 @@ def raisealert(severity, msg, process_name=None):
logging.info(log) logging.info(log)
msg = 'logger -t monit '+ log msg = 'logger -t monit '+ log
pout = Popen(msg, shell=True, stdout=PIPE) pout = Popen(msg, shell=True, stdout=PIPE)
print "[Alert] " + msg print("[Alert] " + msg)
def isPidMatchPidFile(pidfile, pids): def isPidMatchPidFile(pidfile, pids):
@ -148,7 +148,7 @@ def checkProcessRunningStatus(process_name, pidFile):
#cmd = 'service ' + process_name + ' status' #cmd = 'service ' + process_name + ' status'
pout = Popen(cmd, shell=True, stdout=PIPE) pout = Popen(cmd, shell=True, stdout=PIPE)
exitStatus = pout.wait() exitStatus = pout.wait()
temp_out = pout.communicate()[0] temp_out = pout.communicate()[0].decode()
#check there is only one pid or not #check there is only one pid or not
if exitStatus == 0: if exitStatus == 0:
@ -258,12 +258,12 @@ def monitProcess( processes_info ):
printd("No config items provided - means a redundant VR or a VPC Router") printd("No config items provided - means a redundant VR or a VPC Router")
return service_status, failing_services return service_status, failing_services
print "[Process Info] " + json.dumps(processes_info) print("[Process Info] " + json.dumps(processes_info))
#time for noting process down time #time for noting process down time
csec = repr(time.time()).split('.')[0] csec = repr(time.time()).split('.')[0]
for process,properties in processes_info.items(): for process,properties in list(processes_info.items()):
printd ("---------------------------\nchecking the service %s\n---------------------------- " %process) printd ("---------------------------\nchecking the service %s\n---------------------------- " %process)
serviceName = process + ".service" serviceName = process + ".service"
processStatus, wasRestarted = checkProcessStatus(properties) processStatus, wasRestarted = checkProcessStatus(properties)
@ -296,7 +296,7 @@ def execute(script, checkType = "basic"):
pout = Popen(cmd, shell=True, stdout=PIPE) pout = Popen(cmd, shell=True, stdout=PIPE)
exitStatus = pout.wait() exitStatus = pout.wait()
output = pout.communicate()[0].strip() output = pout.communicate()[0].decode().strip()
checkEndTime = time.time() checkEndTime = time.time()
if exitStatus == 0: if exitStatus == 0:

16
systemvm/patch-sysvms.sh
View File

@ -104,16 +104,18 @@ cleanup_systemVM() {
rm -rf $backupfolder rm -rf $backupfolder
mv "$newpath"cloud-scripts.tgz /usr/share/cloud/cloud-scripts.tgz mv "$newpath"cloud-scripts.tgz /usr/share/cloud/cloud-scripts.tgz
rm -rf "$newpath""agent.zip" "$newpath""patch-sysvms.sh" rm -rf "$newpath""agent.zip" "$newpath""patch-sysvms.sh"
if [ "$TYPE" != "consoleproxy" ] && [ "$TYPE" != "secstorage" ]; then
rm -rf /usr/local/cloud/systemvm/
fi
} }
patch_systemvm() { patch_systemvm() {
rm -rf /usr/local/cloud/systemvm rm -rf /usr/local/cloud/systemvm
if [ "$TYPE" == "consoleproxy" ] || [ "$TYPE" == "secstorage" ]; then echo "All" | unzip $newpath/agent.zip -d /usr/local/cloud/systemvm >> $logfile 2>&1
echo "All" | unzip $newpath/agent.zip -d /usr/local/cloud/systemvm >> $logfile 2>&1 mkdir -p /usr/local/cloud/systemvm
mkdir -p /usr/local/cloud/systemvm find /usr/local/cloud/systemvm/ -name \*.sh | xargs chmod 555
find /usr/local/cloud/systemvm/ -name \*.sh | xargs chmod 555
fi
echo "Extracting cloud scripts" >> $logfile 2>&1 echo "Extracting cloud scripts" >> $logfile 2>&1
tar -xvf $newpath/cloud-scripts.tgz -C / >> $logfile 2>&1 tar -xvf $newpath/cloud-scripts.tgz -C / >> $logfile 2>&1
@ -124,6 +126,10 @@ patch_systemvm() {
update_checksum $newpath/cloud-scripts.tgz update_checksum $newpath/cloud-scripts.tgz
if [ -f /opt/cloud/bin/setup/patch.sh ];then
. /opt/cloud/bin/setup/patch.sh && patch_system_vm
fi
if [ "$TYPE" == "consoleproxy" ] || [ "$TYPE" == "secstorage" ] || [[ "$TYPE" == *router ]]; then if [ "$TYPE" == "consoleproxy" ] || [ "$TYPE" == "secstorage" ] || [[ "$TYPE" == *router ]]; then
restart_services restart_services
fi fi

7
systemvm/systemvm-agent-descriptor.xml
View File

@ -121,5 +121,12 @@
<include>**/*</include> <include>**/*</include>
</includes> </includes>
</fileSet> </fileSet>
<fileSet>
<directory>agent/packages</directory>
<outputDirectory>packages</outputDirectory>
<includes>
<include>**/*</include>
</includes>
</fileSet>
</fileSets> </fileSets>
</assembly> </assembly>

20
systemvm/test/__init__.py Executable file
View File

@ -0,0 +1,20 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), "..", "debian/opt/cloud/bin"))

2
systemvm/test/runtests.sh
View File

@ -45,5 +45,5 @@ then
fi fi
echo "Running systemvm/python unit tests" echo "Running systemvm/python unit tests"
nosetests2.7 . nosetests3 .
exit $? exit $?

9
test/integration/smoke/test_network_ipv6.py
View File

@ -697,15 +697,16 @@ class TestIpv6Network(cloudstackTestCase):
"IPv6 firewall rule ICMP code mismatch %d, %d" % (rule.icmpcode, icmp_code)) "IPv6 firewall rule ICMP code mismatch %d, %d" % (rule.icmpcode, icmp_code))
routerCmd = "nft list chain ip6 %s %s" % (FIREWALL_TABLE, FIREWALL_CHAINS[traffic_type]) routerCmd = "nft list chain ip6 %s %s" % (FIREWALL_TABLE, FIREWALL_CHAINS[traffic_type])
res = self.getRouterProcessStatus(self.getNetworkRouter(self.network), routerCmd) res = self.getRouterProcessStatus(self.getNetworkRouter(self.network), routerCmd)
self.assertTrue(parsed_rule in res, parsed_rule_new = parsed_rule.replace("{ ", "").replace(" }", "")
"Listing firewall rule with nft list chain failure for rule: %s" % parsed_rule) self.assertTrue(parsed_rule in res or parsed_rule_new in res,
"Listing firewall rule with nft list chain failure for rule: '%s' is not in '%s'" % (parsed_rule, res))
if delete == True: if delete == True:
cmd = deleteIpv6FirewallRule.deleteIpv6FirewallRuleCmd() cmd = deleteIpv6FirewallRule.deleteIpv6FirewallRuleCmd()
cmd.id = fw_rule.id cmd.id = fw_rule.id
self.userapiclient.deleteIpv6FirewallRule(cmd) self.userapiclient.deleteIpv6FirewallRule(cmd)
res = self.getRouterProcessStatus(self.getNetworkRouter(self.network), routerCmd) res = self.getRouterProcessStatus(self.getNetworkRouter(self.network), routerCmd)
self.assertFalse(parsed_rule in res, self.assertFalse(parsed_rule in res or parsed_rule_new in res,
"Firewall rule present in nft list chain failure despite delete for rule: %s" % parsed_rule) "Firewall rule present in nft list chain failure despite delete for rule: '%s' is in '%s'" % (parsed_rule, res))
def checkIpv6FirewallRule(self): def checkIpv6FirewallRule(self):
traffic_type = "Ingress" traffic_type = "Ingress"

79
test/integration/smoke/test_routers.py
View File

@ -22,7 +22,8 @@ from marvin.cloudstackTestCase import cloudstackTestCase
from marvin.cloudstackAPI import (stopRouter, from marvin.cloudstackAPI import (stopRouter,
restartNetwork, restartNetwork,
startRouter, startRouter,
rebootRouter) rebootRouter,
getRouterHealthCheckResults)
from marvin.lib.utils import (cleanup_resources, from marvin.lib.utils import (cleanup_resources,
get_process_status, get_process_status,
get_host_credentials) get_host_credentials)
@ -303,7 +304,81 @@ class TestRouterServices(cloudstackTestCase):
"Check haproxy service is running or not" "Check haproxy service is running or not"
) )
self.debug("Haproxy process status: %s" % res) self.debug("Haproxy process status: %s" % res)
return
routers = list_routers(
self.apiclient,
account=self.account.name,
domainid=self.account.domainid,
fetchhealthcheckresults=True
)
self.assertEqual(isinstance(routers, list), True,
"Check for list routers response return valid data"
)
self.assertNotEqual(
len(routers), 0,
"Check list router response"
)
router = routers[0]
self.info("Router ID: %s & Router state: %s" % (
router.id, router.state
))
self.assertEqual(isinstance(router.healthcheckresults, list), True,
"Router response should contain it's health check result as list"
)
cmd = getRouterHealthCheckResults.getRouterHealthCheckResultsCmd()
cmd.routerid = router.id
cmd.performfreshchecks = True # Perform fresh checks as a newly created router may not have results
healthData = self.apiclient.getRouterHealthCheckResults(cmd)
self.info("Router ID: %s & Router state: %s" % (
router.id, router.state
))
self.assertEqual(router.id, healthData.routerid,
"Router response should contain it's health check result so id should match"
)
self.assertEqual(isinstance(healthData.healthchecks, list), True,
"Router response should contain it's health check result as list"
)
self.verifyCheckTypes(healthData.healthchecks)
self.verifyCheckNames(healthData.healthchecks)
self.verifyCheckResults(healthData.healthchecks)
def verifyCheckTypes(self, healthChecks):
for checkType in ["basic", "advanced"]:
foundType = False
for check in healthChecks:
if check.checktype == checkType:
foundType = True
break
self.assertTrue(foundType,
"Router should contain health check results info for type: " + checkType
)
def verifyCheckNames(self, healthChecks):
for checkName in ["dns_check.py", "dhcp_check.py", "haproxy_check.py", "disk_space_check.py", "iptables_check.py", "gateways_check.py", "router_version_check.py"]:
foundCheck = False
for check in healthChecks:
if check.checkname == checkName:
foundCheck = True
break
self.assertTrue(foundCheck,
"Router should contain health check results info for check name: " + checkName
)
def verifyCheckResults(self, healthChecks):
failedCheck = 0
for check in healthChecks:
if check.success:
print("check %s is good" % check.checkname)
else:
print("check %s failed due to %s" % (check.checkname, check.details))
failedCheck = failedCheck + 1
self.assertEquals(failedCheck, 0)
@attr( @attr(
tags=[ tags=[

32
test/integration/smoke/test_routers_network_ops.py
View File

@ -287,8 +287,8 @@ class TestRedundantIsolateNetworks(cloudstackTestCase):
) )
expected = 1 expected = 1
ssh_command = "wget -t 1 -T 5 www.google.com" ssh_command = "curl -v -m 1 -o index.html -sL www.google.com"
check_string = "HTTP request sent, awaiting response... 200 OK" check_string = "200 OK"
result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self) result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self)
self.assertEqual( self.assertEqual(
@ -307,8 +307,8 @@ class TestRedundantIsolateNetworks(cloudstackTestCase):
) )
expected = 0 expected = 0
ssh_command = "wget -t 1 -T 1 www.google.com" ssh_command = "curl -v -m 1 -o index.html -sL www.google.com"
check_string = "HTTP request sent, awaiting response... 200 OK" check_string = "200 OK"
result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self) result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self)
self.assertEqual( self.assertEqual(
@ -451,8 +451,8 @@ class TestRedundantIsolateNetworks(cloudstackTestCase):
) )
expected = 0 expected = 0
ssh_command = "wget -t 1 -T 1 www.google.com" ssh_command = "curl -v -m 1 -o index.html -sL www.google.com"
check_string = "HTTP request sent, awaiting response... 200 OK" check_string = "200 OK"
result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self) result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self)
self.assertEqual( self.assertEqual(
@ -480,8 +480,8 @@ class TestRedundantIsolateNetworks(cloudstackTestCase):
) )
expected = 1 expected = 1
ssh_command = "wget -t 1 -T 5 www.google.com" ssh_command = "curl -v -m 1 -o index.html -sL www.google.com"
check_string = "HTTP request sent, awaiting response... 200 OK" check_string = "200 OK"
result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self) result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self)
self.assertEqual( self.assertEqual(
@ -840,8 +840,8 @@ class TestIsolatedNetworks(cloudstackTestCase):
) )
expected = 1 expected = 1
ssh_command = "wget -t 1 -T 5 www.google.com" ssh_command = "curl -v -m 1 -o index.html -sL www.google.com"
check_string = "HTTP request sent, awaiting response... 200 OK" check_string = "200 OK"
result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self) result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self)
self.assertEqual( self.assertEqual(
@ -860,8 +860,8 @@ class TestIsolatedNetworks(cloudstackTestCase):
) )
expected = 0 expected = 0
ssh_command = "wget -t 1 -T 1 www.google.com" ssh_command = "curl -v -m 1 -o index.html -sL www.google.com"
check_string = "HTTP request sent, awaiting response... 200 OK" check_string = "200 OK"
result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self) result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self)
self.assertEqual( self.assertEqual(
@ -995,8 +995,8 @@ class TestIsolatedNetworks(cloudstackTestCase):
) )
expected = 0 expected = 0
ssh_command = "wget -t 1 -T 1 www.google.com" ssh_command = "curl -v -m 1 -o index.html -sL www.google.com"
check_string = "HTTP request sent, awaiting response... 200 OK" check_string = "200 OK"
result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self) result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self)
self.assertEqual( self.assertEqual(
@ -1015,8 +1015,8 @@ class TestIsolatedNetworks(cloudstackTestCase):
) )
expected = 1 expected = 1
ssh_command = "wget -t 1 -T 5 www.google.com" ssh_command = "curl -v -m 1 -o index.html -sL www.google.com"
check_string = "HTTP request sent, awaiting response... 200 OK" check_string = "200 OK"
result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self) result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self)
self.assertEqual( self.assertEqual(

5
test/integration/smoke/test_vpc_ipv6.py
View File

@ -761,8 +761,9 @@ class TestIpv6Vpc(cloudstackTestCase):
acl_chain = nic + ACL_CHAINS_SUFFIX[rule["traffictype"]] acl_chain = nic + ACL_CHAINS_SUFFIX[rule["traffictype"]]
routerCmd = "nft list chain ip6 %s %s" % (ACL_TABLE, acl_chain) routerCmd = "nft list chain ip6 %s %s" % (ACL_TABLE, acl_chain)
res = self.getRouterProcessStatus(router, routerCmd) res = self.getRouterProcessStatus(router, routerCmd)
self.assertTrue(rule["parsedrule"] in res, parsed_rule_new = rule["parsedrule"].replace("{ ", "").replace(" }", "")
"Listing firewall rule with nft list chain failure for rule: %s" % rule["parsedrule"]) self.assertTrue(rule["parsedrule"] in res or parsed_rule_new in res,
"Listing firewall rule with nft list chain failure for rule: '%s' is not in '%s'" % (rule["parsedrule"], res))
def checkIpv6AclRule(self): def checkIpv6AclRule(self):
router = self.getVpcRouter(self.vpc) router = self.getVpcRouter(self.vpc)

2
test/integration/smoke/test_vpc_vpn.py
View File

@ -592,7 +592,7 @@ class TestVpcSite2SiteVpn(cloudstackTestCase):
time.sleep(20) time.sleep(20)
# setup ssh connection to vm2 # setup ssh connection to vm2
ssh_client = self._get_ssh_client(vm2, self.services, 10) ssh_client = self._get_ssh_client(vm2, self.services, 30)
if ssh_client: if ssh_client:
# run ping test # run ping test

2
tools/appliance/systemvmtemplate/http/preseed.cfg
View File

@ -66,7 +66,7 @@ d-i partman-auto/expert_recipe string \
use_filesystem{ } filesystem{ ext2 } \ use_filesystem{ } filesystem{ ext2 } \
mountpoint{ /boot } \ mountpoint{ /boot } \
. \ . \
256 1000 256 linux-swap \ 512 1000 512 linux-swap \
method{ swap } format{ } \ method{ swap } format{ } \
. \ . \
2240 40 4000 ext4 \ 2240 40 4000 ext4 \

4
tools/appliance/systemvmtemplate/scripts/apt_upgrade.sh
View File

@ -36,8 +36,8 @@ function add_backports() {
sed -i '/deb-src/d' /etc/apt/sources.list sed -i '/deb-src/d' /etc/apt/sources.list
sed -i '/backports/d' /etc/apt/sources.list sed -i '/backports/d' /etc/apt/sources.list
sed -i '/security/d' /etc/apt/sources.list sed -i '/security/d' /etc/apt/sources.list
echo 'deb http://http.debian.net/debian bullseye-backports main' >> /etc/apt/sources.list echo 'deb http://http.debian.net/debian bookworm-backports main' >> /etc/apt/sources.list
echo 'deb http://security.debian.org/debian-security bullseye-security main' >> /etc/apt/sources.list echo 'deb http://security.debian.org/debian-security bookworm-security main' >> /etc/apt/sources.list
} }
function apt_upgrade() { function apt_upgrade() {

2
tools/appliance/systemvmtemplate/scripts/configure_systemvm_services.sh
View File

@ -19,7 +19,7 @@
set -e set -e
set -x set -x
CLOUDSTACK_RELEASE=4.19.0 CLOUDSTACK_RELEASE=4.20.0
function configure_apache2() { function configure_apache2() {
# Enable ssl, rewrite and auth # Enable ssl, rewrite and auth

18
tools/appliance/systemvmtemplate/scripts/install_systemvm_packages.sh
View File

@ -22,7 +22,7 @@ set -x
function install_vhd_util() { function install_vhd_util() {
[[ -f /bin/vhd-util ]] && return [[ -f /bin/vhd-util ]] && return
wget --no-check-certificate https://github.com/shapeblue/cloudstack-nonoss/raw/main/vhd-util -O /bin/vhd-util wget --no-check-certificate https://download.cloudstack.org/tools/vhd-util -O /bin/vhd-util
chmod a+x /bin/vhd-util chmod a+x /bin/vhd-util
} }
@ -53,7 +53,7 @@ function install_packages() {
${apt_get} install grub-legacy \ ${apt_get} install grub-legacy \
rsyslog logrotate cron net-tools ifupdown tmux vim-tiny htop netbase iptables nftables \ rsyslog logrotate cron net-tools ifupdown tmux vim-tiny htop netbase iptables nftables \
openssh-server e2fsprogs tcpdump iftop socat wget coreutils systemd \ openssh-server e2fsprogs tcpdump iftop socat wget coreutils systemd \
python python3 python3-flask ieee-data \ python-is-python3 python3 python3-flask python3-netaddr ieee-data \
bzip2 sed gawk diffutils grep gzip less tar telnet ftp rsync traceroute psmisc lsof procps \ bzip2 sed gawk diffutils grep gzip less tar telnet ftp rsync traceroute psmisc lsof procps \
inetutils-ping iputils-arping httping curl \ inetutils-ping iputils-arping httping curl \
dnsutils zip unzip ethtool uuid file iproute2 acpid sudo \ dnsutils zip unzip ethtool uuid file iproute2 acpid sudo \
@ -63,10 +63,10 @@ function install_packages() {
nfs-common \ nfs-common \
samba-common cifs-utils \ samba-common cifs-utils \
xl2tpd bcrelay ppp tdb-tools \ xl2tpd bcrelay ppp tdb-tools \
xenstore-utils libxenstore3.0 \ xenstore-utils libxenstore4 \
ipvsadm conntrackd libnetfilter-conntrack3 \ ipvsadm conntrackd libnetfilter-conntrack3 \
keepalived irqbalance \ keepalived irqbalance \
openjdk-11-jre-headless \ openjdk-17-jre-headless \
ipcalc ipset \ ipcalc ipset \
iptables-persistent \ iptables-persistent \
libtcnative-1 libssl-dev libapr1-dev \ libtcnative-1 libssl-dev libapr1-dev \
@ -80,10 +80,6 @@ function install_packages() {
apt-get install -y python3-json-pointer python3-jsonschema cloud-init apt-get install -y python3-json-pointer python3-jsonschema cloud-init
# python2-netaddr workaround
wget https://github.com/shapeblue/cloudstack-nonoss/raw/main/python-netaddr_0.7.19-1_all.deb
dpkg -i python-netaddr_0.7.19-1_all.deb
apt_clean apt_clean
# 32 bit architecture support for vhd-util # 32 bit architecture support for vhd-util
@ -104,9 +100,9 @@ function install_packages() {
install_vhd_util install_vhd_util
# Install xenserver guest utilities as debian repos don't have it # Install xenserver guest utilities as debian repos don't have it
wget https://mirrors.kernel.org/ubuntu/pool/main/x/xe-guest-utilities/xe-guest-utilities_7.10.0-0ubuntu1_amd64.deb wget --no-check-certificate https://download.cloudstack.org/systemvm/debian/xe-guest-utilities_7.20.2-0ubuntu1_amd64.deb
dpkg -i xe-guest-utilities_7.10.0-0ubuntu1_amd64.deb dpkg -i xe-guest-utilities_7.20.2-0ubuntu1_amd64.deb
rm -f xe-guest-utilities_7.10.0-0ubuntu1_amd64.deb rm -f xe-guest-utilities_7.20.2-0ubuntu1_amd64.deb
} }
return 2>/dev/null || install_packages return 2>/dev/null || install_packages

4
tools/appliance/systemvmtemplate/template.json
View File

@ -27,8 +27,8 @@
"format": "qcow2", "format": "qcow2",
"headless": true, "headless": true,
"http_directory": "http", "http_directory": "http",
"iso_checksum": "sha512:da7e7867ed043b784f5ae7e4adaaf4f023b5235f0fa2ead1279dc93f74bc17801ed906d330e3cd68ee8d3e96b697d21d23cfe2b755f5a9eb555bd5390a8c4dac", "iso_checksum": "sha512:33c08e56c83d13007e4a5511b9bf2c4926c4aa12fd5dd56d493c0653aecbab380988c5bf1671dbaea75c582827797d98c4a611f7fb2b131fbde2c677d5258ec9",
"iso_url": "https://cdimage.debian.org/mirror/cdimage/archive/11.8.0/amd64/iso-cd/debian-11.8.0-amd64-netinst.iso", "iso_url": "https://download.cloudstack.org/systemvm/debian/debian-12.5.0-amd64-netinst.iso",
"net_device": "virtio-net", "net_device": "virtio-net",
"output_directory": "../dist", "output_directory": "../dist",
"qemuargs": [ "qemuargs": [

10
vmware-base/pom.xml
View File

@ -51,6 +51,16 @@
<groupId>com.google.code.gson</groupId> <groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId> <artifactId>gson</artifactId>
</dependency> </dependency>
<dependency>
<groupId>jakarta.xml.bind</groupId>
<artifactId>jakarta.xml.bind-api</artifactId>
<version>${cs.jakarta.xml.bind.version}</version>
</dependency>
<dependency>
<groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId>
<version>${cs.jaxb.impl.version}</version>
</dependency>
<dependency> <dependency>
<groupId>com.cloud.com.vmware</groupId> <groupId>com.cloud.com.vmware</groupId>
<artifactId>vmware-vim25</artifactId> <artifactId>vmware-vim25</artifactId>