29 Commits

Author SHA1 Message Date
Wei Zhou
9f5ac89c9a
VR: fix data-server if shared network has multiple ip ranges (#5530)
* VR: fix data-server if shared network has multiple ip ranges

This fixes #5518

* Update PR #5530: fix nameserver in vm with IP in additional IP ranges

without this change
```
root@r-757-VM:~# cat /etc/dnsmasq.d/cloud.conf
listen-address=127.0.0.1,10.10.12.31,10.10.13.19
dhcp-range=set:interface-eth0-0,10.10.12.31,static
dhcp-option=tag:interface-eth0-0,15,cs1cloud.internal
dhcp-option=tag:interface-eth0-0,6,10.10.12.31,10.0.32.1,8.8.8.8
dhcp-option=tag:interface-eth0-0,3,10.10.12.254
dhcp-option=tag:interface-eth0-0,1,255.255.255.0
dhcp-range=set:interface-eth0-1,10.10.13.19,static
dhcp-option=tag:interface-eth0-1,15,cs1cloud.internal
dhcp-option=tag:interface-eth0-1,6,10.10.12.31,10.0.32.1,8.8.8.8 <<< nameserver 10.10.12.31
dhcp-option=tag:interface-eth0-1,3,10.10.13.254
dhcp-option=tag:interface-eth0-1,1,255.255.255.0
```

with this change
```
root@r-757-VM:~# cat /etc/dnsmasq.d/cloud.conf
listen-address=127.0.0.1,10.10.12.31,10.10.13.19
dhcp-range=set:interface-eth0-0,10.10.12.31,static
dhcp-option=tag:interface-eth0-0,15,cs1cloud.internal
dhcp-option=tag:interface-eth0-0,6,10.10.12.31,10.0.32.1,8.8.8.8
dhcp-option=tag:interface-eth0-0,3,10.10.12.254
dhcp-option=tag:interface-eth0-0,1,255.255.255.0
dhcp-range=set:interface-eth0-1,10.10.13.19,static
dhcp-option=tag:interface-eth0-1,15,cs1cloud.internal
dhcp-option=tag:interface-eth0-1,6,10.10.13.19,10.0.32.1,8.8.8.8 <<< nameserver 10.10.13.19
dhcp-option=tag:interface-eth0-1,3,10.10.13.254
dhcp-option=tag:interface-eth0-1,1,255.255.255.0
```

* Update #5530: add 'localise-queries' to /etc/dnsmasq.conf
2021-10-04 11:40:25 +02:00
Rohit Yadav
57a55bd777 Merge remote-tracking branch 'origin/4.15' into main 2021-08-20 17:19:40 +05:30
Wei Zhou
0390281ad2
vr: cleanup files in /var/cache/cloud/processed every day (#5331)
* vr: cleanup files in /var/cache/cloud/processed every day

* #5331: add license header
2021-08-20 17:10:04 +05:30
Rohit Yadav
69a0ab665d
systemvmtemplate: bump to Debian 11.0.0 systemvmtemplate (#5317)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-08-18 16:47:08 +05:30
Rohit Yadav
e824fdba20 Merge remote-tracking branch 'origin/4.14' into 4.15
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Conflicts:
	server/src/main/java/com/cloud/vm/UserVmManagerImpl.java
2021-04-10 13:41:50 +05:30
Spaceman1984
4bab06a74b
systemvm: Restricting http access on VR to internal network (#4847)
There is a potential security issue with having http access to the VR from anywhere.
This PR restricts http access to the VR to the internal network only.
2021-04-10 13:19:31 +05:30
Rohit Yadav
43257f8300 Merge remote-tracking branch 'origin/4.14' into 4.15 2021-04-05 12:59:37 +05:30
aleskxyz
ca4669c4d4
systemvm: Add localized "data-server" records in /etc/hosts for VPC routers (#4873) 2021-04-05 12:34:10 +05:30
Wei Zhou
d4ba00434c
VR: fix rsyslog compresses log files but not release disk space in VR (#4869)
We had critical issue with VR recently. The VRs of shared network or vpc stops working after some days.
After investigation, I found that the disk space is full

```
root@r-10-VM:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda5        2086316 2069932         0 100% /
```

logrotate/ryslog has compresses the log files, but space is not released. see `lsof |grep deleted`

```
root@r-10-VM:~# lsof |grep deleted
rsyslogd    960                      root   12w      REG              254,5 493060096        137 /var/log/daemon.log.1 (deleted)
rsyslogd    960                      root   13w      REG              254,5  17715200        110 /var/log/messages.1 (deleted)
rsyslogd    960                      root   16w      REG              254,5 545968128        342 /var/log/auth.log.1 (deleted)
rsyslogd    960                      root   18w      REG              254,5  38313984        341 /var/log/cron.log.1 (deleted)
rsyslogd    960  962 in:imuxso       root   12w      REG              254,5 493060096        137 /var/log/daemon.log.1 (deleted)
rsyslogd    960  962 in:imuxso       root   13w      REG              254,5  17715200        110 /var/log/messages.1 (deleted)
rsyslogd    960  962 in:imuxso       root   16w      REG              254,5 545968128        342 /var/log/auth.log.1 (deleted)
rsyslogd    960  962 in:imuxso       root   18w      REG              254,5  38313984        341 /var/log/cron.log.1 (deleted)
rsyslogd    960  963 in:imklog       root   12w      REG              254,5 493060096        137 /var/log/daemon.log.1 (deleted)
rsyslogd    960  963 in:imklog       root   13w      REG              254,5  17715200        110 /var/log/messages.1 (deleted)
rsyslogd    960  963 in:imklog       root   16w      REG              254,5 545968128        342 /var/log/auth.log.1 (deleted)
rsyslogd    960  963 in:imklog       root   18w      REG              254,5  38313984        341 /var/log/cron.log.1 (deleted)
rsyslogd    960  964 in:imfile       root   12w      REG              254,5 493060096        137 /var/log/daemon.log.1 (deleted)
rsyslogd    960  964 in:imfile       root   13w      REG              254,5  17715200        110 /var/log/messages.1 (deleted)
rsyslogd    960  964 in:imfile       root   16w      REG              254,5 545968128        342 /var/log/auth.log.1 (deleted)
rsyslogd    960  964 in:imfile       root   18w      REG              254,5  38313984        341 /var/log/cron.log.1 (deleted)
rsyslogd    960  965 in:imudp        root   12w      REG              254,5 493060096        137 /var/log/daemon.log.1 (deleted)
rsyslogd    960  965 in:imudp        root   13w      REG              254,5  17715200        110 /var/log/messages.1 (deleted)
rsyslogd    960  965 in:imudp        root   16w      REG              254,5 545968128        342 /var/log/auth.log.1 (deleted)
rsyslogd    960  965 in:imudp        root   18w      REG              254,5  38313984        341 /var/log/cron.log.1 (deleted)
rsyslogd    960  966 rs:main         root   12w      REG              254,5 493060096        137 /var/log/daemon.log.1 (deleted)
rsyslogd    960  966 rs:main         root   13w      REG              254,5  17715200        110 /var/log/messages.1 (deleted)
rsyslogd    960  966 rs:main         root   16w      REG              254,5 545968128        342 /var/log/auth.log.1 (deleted)
rsyslogd    960  966 rs:main         root   18w      REG              254,5  38313984        341 /var/log/cron.log.1 (deleted)
```

workaround: restarting rsyslog to release the space.
```
systemctl restart rsyslog
```

The root cause is, the following command does not work in 4.15 template
```
root@r-10-VM:~# invoke-rc.d rsyslog rotate
[FAIL] Closing open files: rsyslogd failed!
```

Fix: use `/usr/lib/rsyslog/rsyslog-rotate` instead
```
root@r-10-VM:~# /usr/lib/rsyslog/rsyslog-rotate
root@r-10-VM:~# cat /usr/lib/rsyslog/rsyslog-rotate

if [ -d /run/systemd/system ]; then
    systemctl kill -s HUP rsyslog.service
else
    invoke-rc.d rsyslog rotate > /dev/null
fi

```
2021-04-01 14:30:58 +05:30
Wei Zhou
dc5b9ec7c8
systemvm: remove logrotate config for wtmp and btmp (#4872)
logrotate in systemvms run every day. it exits with failure.
```
root@r-100-VM:~# systemctl status logrotate
● logrotate.service - Rotate log files
   Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-03-23 00:00:01 UTC; 2 days ago
     Docs: man:logrotate(8)
           man:logrotate.conf(5)
  Process: 25001 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
 Main PID: 25001 (code=exited, status=1/FAILURE)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
```

it is because the logrotate for wtmp and btmp already exist in 4.15 systemvm template.
```
root@r-100-VM:~# cat /etc/logrotate.d/wtmp
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    minsize 1M
    rotate 1
}
root@r-100-VM:~# cat /etc/logrotate.d/btmp
/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}
```

remove them from /etc/logrotate.conf fixes the issue.
```
root@r-100-VM:~# systemctl status logrotate
● logrotate.service - Rotate log files
   Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
   Active: inactive (dead) since Thu 2021-03-25 00:00:01 UTC; 9h ago
     Docs: man:logrotate(8)
           man:logrotate.conf(5)
  Process: 28211 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=0/SUCCESS)
 Main PID: 28211 (code=exited, status=0/SUCCESS)

Mar 25 00:00:01 r-100-VM systemd[1]: Starting Rotate log files...
Mar 25 00:00:01 r-100-VM systemd[1]: logrotate.service: Succeeded.
Mar 25 00:00:01 r-100-VM systemd[1]: Started Rotate log files.
```
2021-04-01 12:51:17 +05:30
Spaceman1984
88762c101c
Added compress option to dnsmasq log files (#4439) 2020-11-06 09:33:52 +00:00
Rohit Yadav
1e19ea5bdd
systemvmtemplate: move to using Debian10 (#4104)
This upgrades the systemvmtemplate base to Debian 10 with openjdk-11 and a newer strongswan package.

Fixes #3654

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-06-09 08:20:51 +05:30
davidjumani
1756b0f64a
noVNC console integration (#3967)
* Adding noVNC repo

* Adding support for noVNC

* Adding Ctl+Esc

* Removing device name from novnc header
2020-05-19 14:14:04 +02:00
Daan Hoogland
06a8ff04b1 Merge release branch 4.13 to master
* 4.13:
  VR: Fix Redundant VRouter guest network on wrong interface (#3847)
2020-02-29 19:56:07 +01:00
Wei Zhou
313e21a0da
VR: Fix Redundant VRouter guest network on wrong interface (#3847) 2020-02-29 19:52:40 +01:00
Rohit Yadav
d90341ebf1
cloudstack: add JDK11 support (#3601)
This adds support for JDK11 in CloudStack 4.14+:

- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-02-12 12:58:25 +05:30
Anurag Awasthi
c0abfce8fa
Health check feature for virtual router (#3575) 2020-01-30 12:39:03 +01:00
Wei Zhou
ff1c6e78f4 router: Set up metadata/password/dhcp server on gateway IP instead of guest IP in RVR (#3477)
When we create a vm in the network with redundant VRs, the lease file in the vm (for example /var/lib/dhcp/dhclient.eth0.leases) shows the dhcp-server-identifier is the guest ip (not vip/gateway) of master VR. That's the ip ipaddress where the vm fetch password and metadata from.
if we stop the master VR (then backup will be master) or restart the network with cleanup (VRs will be created), the guest ip of master VR changes so vm are not able to get metadata/ssh-key using the ips in dhcp lease file.

Setting up metadata/password/dhcp server on gateway instead of guest IP in redundant VRs will fix the issues.

FIxes #3409
2020-01-28 10:35:59 +05:30
Paul Angus
033199972e systemvm: improve SystemVM startup and memory usage (#3126)
In order to reduce memory footprint and improve boot speed/predictability.
The following changes have been made:

- add vm.min_free_kbytes to sysctl
- periodically clear disk cache (depending on memory size)
- only start guest services specific to hypervisor
- use systemvm code to determine hypervisor type (not systemd)
- start cloud service at end of post init rather than through systemd
- reduce initial threads started for httpd
- fix vmtools config file

Fixes #3039

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-06-26 14:40:59 +05:30
Rohit Yadav
e092529c98
systemvm: Ensure cloud service reboots after failure (#2916)
This fixes an issue for systemvms (CPVM and SSVM) on VMware, as eth0
is not programmed (link-local) the networking.service fails to start
which is a dependency for cloud-postinit service. When cloud-postinit
service fails to start/run, it fails to start the agent (cloud) process.
This fixes the smoketest failures we saw in case of VMware 6.5 with
4.11.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-23 23:33:08 +05:30
Rohit Yadav
933ee23104
vr: memory and swap optimizations (#2892)
This tries to provide a threshold based fix for #2873 where swappinness of VR is not used until last resort. By limiting swappiness unless actually needed, the VR system degradation can be avoided for most cases. The other change is around not starting baremetal-vr by default on all VRs, according to the spec https://cwiki.apache.org/confluence/display/CLOUDSTACK/Baremetal+Advanced+Networking+Support only vmware VRs need to run it and that too only as the last step of the setup/completion, so we don't need to run it all the time.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-16 10:29:48 +05:30
Rohit Yadav
464551208c
xenserver: Add support for XS 7.3, 7.4 and XCP-ng 7.4 (#2605)
This adds support for XenServer 7.3 and 7.4, and XCP-ng 7.4 version as hypervisor hosts. Fixes #2523.

This also fixes the issue of 4.11 VRs stuck in starting for up-to 10mins, before they come up online.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-04-30 08:19:10 +02:00
Rohit Yadav
e71d4d4371 CLOUDSTACK-10304: turn off apache2 server tokens and signature in systemvms (#2563)
* systemvm: turn off apache2 server tokens and signature

This turns off apache2 server version signature/token in headers.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* systemvm: remove invalid code as conf.d is not available now

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-04-13 10:44:25 +02:00
René Moser
c8dcc64b65 CLOUDSTACK-10341: VR minor fixes to systemvmtemplate (#2468)
- Fixes rsyslog: fix config error in rsylslog.conf

Feb 26 08:09:54 r-413-VM liblogging-stdlog[19754]: action '*' treated as ':omusrmsg:*' - please use ':omusrmsg:*' syntax instead, '*' will not be supported in the future [v8.24.0 try http://www.rsyslog.com/e/2184 ]
Feb 26 08:09:54 r-413-VM liblogging-stdlog[19754]: error during parsing file /etc/rsyslog.conf, on or before line 95: warnings occured in file '/etc/rsyslog.conf' around line 95 [v8.24.0 try http://www.rsyslog.com/e/2207 ]

- Run apache2 only after cloud-postinit

- Increase /run size for VR with 256M RAM

root@r-395-VM:~# systemctl daemon-reload
Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd. Currently, 15.8M are free, but a safety buffer of 16.0M is enforced.

tmpfs            23M  6.5M   16M  29% /run
2018-03-23 11:52:29 +05:30
Rohit Yadav
da8cf8c370
CLOUDSTACK-10319: Prefer TLSv1.2, deprecate TLSv1.0,1.1 (#2480)
This deprecates and remove TLS 1.0 and 1.1 from preferred list of
protocols and keeps only TLSv1.2.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-03-12 11:43:59 +01:00
Rohit Yadav
d19629a115 CLOUDSTACK-10013: Fixes based on code review and test failures
This includes test related fixes and code review fixes based on
reviews from @rafaelweingartner, @marcaurele, @wido and @DaanHoogland.

This also includes VMware disk-resize limitation bug fix based on comments
from @sateesh-chodapuneedi and @priyankparihar.

This also includes the final changes to systemvmtemplate and fixes to
code based on issues found via test failures.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 17:51:42 +05:30
Rohit Yadav
15b11a3b27 CLOUDSTACK-10013: Fix VMware related issues and fix misc tests
This fixes test failures around VMware with the new systemvmtemplate.
In addition:

- Does not skip rVR related test cases for VMware
- Removes rc.local
- Processes unprocessed cmd_line.json
- Fixed NPEs around VMware tests/code
- On VMware, use udevadm to reconfigure nic/mac address than rebooting
- Fix proper acpi shutdown script for faster systemvm shutdowns
- Give at least 256MB of swap for VRs to avoid OOM on VMware
- Fixes smoke tests for environment related failures

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Rohit Yadav
facc5945f0 CLOUDSTACK-10193: Fix smoke tests failures with new systemvmtemplate
- Several systemvmtemplate optimizations
- Uses new macchinina template for running smoke tests
- Switch to latest Debian 9.3.0 release for systemvmtemplate
- Introduce a new `get_test_template` that uses tiny test template
  such as macchinina as defined test_data.py
- rVR related fixes and improvements

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Rohit Yadav
85aee8d18d CLOUDSTACK-10013: SystemVM codebase refactorings and improvements
- Refactors and simplifies systemvm codebase file structures keeping
  the same resultant systemvm.iso packaging
- Password server systemd script and new postinit script that runs
  before sshd starts
- Fixes to keepalived and conntrackd config to make rVRs work again
- New /etc/issue featuring ascii based cloudmonkey logo/message and
  systemvmtemplate version
- SystemVM python codebase linted and tested. Added pylint/pep to
  Travis.
- iptables re-application fixes for non-VR systemvms.
- SystemVM template build fixes.
- Default secondary storage vm service offering boosted to have 2vCPUs
  and RAM equal to console proxy.
- Fixes to several marvin based smoke tests, especially rVR related
  tests. rVR tests to consider 3*advert_int+skew timeout before status
  is checked.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30