Wei Zhou
d4ba00434c
VR: fix rsyslog compresses log files but not release disk space in VR ( #4869 )
...
We had critical issue with VR recently. The VRs of shared network or vpc stops working after some days.
After investigation, I found that the disk space is full
```
root@r-10-VM:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda5 2086316 2069932 0 100% /
```
logrotate/ryslog has compresses the log files, but space is not released. see `lsof |grep deleted`
```
root@r-10-VM:~# lsof |grep deleted
rsyslogd 960 root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 962 in:imuxso root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 962 in:imuxso root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 962 in:imuxso root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 962 in:imuxso root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 963 in:imklog root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 963 in:imklog root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 963 in:imklog root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 963 in:imklog root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 964 in:imfile root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 964 in:imfile root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 964 in:imfile root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 964 in:imfile root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 965 in:imudp root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 965 in:imudp root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 965 in:imudp root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 965 in:imudp root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 966 rs:main root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 966 rs:main root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 966 rs:main root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 966 rs:main root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
```
workaround: restarting rsyslog to release the space.
```
systemctl restart rsyslog
```
The root cause is, the following command does not work in 4.15 template
```
root@r-10-VM:~# invoke-rc.d rsyslog rotate
[FAIL] Closing open files: rsyslogd failed!
```
Fix: use `/usr/lib/rsyslog/rsyslog-rotate` instead
```
root@r-10-VM:~# /usr/lib/rsyslog/rsyslog-rotate
root@r-10-VM:~# cat /usr/lib/rsyslog/rsyslog-rotate
if [ -d /run/systemd/system ]; then
systemctl kill -s HUP rsyslog.service
else
invoke-rc.d rsyslog rotate > /dev/null
fi
```
2021-04-01 14:30:58 +05:30
Wei Zhou
dc5b9ec7c8
systemvm: remove logrotate config for wtmp and btmp ( #4872 )
...
logrotate in systemvms run every day. it exits with failure.
```
root@r-100-VM:~# systemctl status logrotate
● logrotate.service - Rotate log files
Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2021-03-23 00:00:01 UTC; 2 days ago
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 25001 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 25001 (code=exited, status=1/FAILURE)
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
```
it is because the logrotate for wtmp and btmp already exist in 4.15 systemvm template.
```
root@r-100-VM:~# cat /etc/logrotate.d/wtmp
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
root@r-100-VM:~# cat /etc/logrotate.d/btmp
/var/log/btmp {
missingok
monthly
create 0660 root utmp
rotate 1
}
```
remove them from /etc/logrotate.conf fixes the issue.
```
root@r-100-VM:~# systemctl status logrotate
● logrotate.service - Rotate log files
Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
Active: inactive (dead) since Thu 2021-03-25 00:00:01 UTC; 9h ago
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 28211 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=0/SUCCESS)
Main PID: 28211 (code=exited, status=0/SUCCESS)
Mar 25 00:00:01 r-100-VM systemd[1]: Starting Rotate log files...
Mar 25 00:00:01 r-100-VM systemd[1]: logrotate.service: Succeeded.
Mar 25 00:00:01 r-100-VM systemd[1]: Started Rotate log files.
```
2021-04-01 12:51:17 +05:30
Rohit Yadav
85aee8d18d
CLOUDSTACK-10013: SystemVM codebase refactorings and improvements
...
- Refactors and simplifies systemvm codebase file structures keeping
the same resultant systemvm.iso packaging
- Password server systemd script and new postinit script that runs
before sshd starts
- Fixes to keepalived and conntrackd config to make rVRs work again
- New /etc/issue featuring ascii based cloudmonkey logo/message and
systemvmtemplate version
- SystemVM python codebase linted and tested. Added pylint/pep to
Travis.
- iptables re-application fixes for non-VR systemvms.
- SystemVM template build fixes.
- Default secondary storage vm service offering boosted to have 2vCPUs
and RAM equal to console proxy.
- Fixes to several marvin based smoke tests, especially rVR related
tests. rVR tests to consider 3*advert_int+skew timeout before status
is checked.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
