前段时间发现某些机器磁盘空间报警,使用 du 命令(慎重使用)查询后发现部分日志文件非常大,例如 secure 日志,差不多有 10G 左右。

$ ls -lh /var/log/secure
-rw-r----- 1 root adm 9.7G Mar 24 20:44 /var/log/secure

我们发现 secure 日志已经超过一周没有滚动了,按照 logrotate 的配置,secure 日志应该按周滚动一次,最多滚动 4 次:

$cat /etc/logrotate.d/syslog-ng
/var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron {
    sharedscripts
    postrotate
        /etc/rc.d/init.d/syslog-ng reload 2>/dev/null || true
    endscript
}

$cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

看起来应该是日志滚动过程出现了问题,然后通过 logroate debug 了一把,发现中间出错:

$ logrotate -dv /etc/logrotate.conf 
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file acpid
reading config info for /var/log/acpid 
reading config file balloond
reading config info for /var/log/xen/balloond.log 
reading config file conman
error: error accessing /var/log/conman: No such file or directory
error: conman:5 glob failed for /var/log/conman/*

看最后两行 error,因为 /var/log/conman 目录找不到,导致滚动过程出错,所以就没有触发 secure 日志滚动处理。

那么最直接的解决方法是,手工创建 /var/log/conman 目录,这个问题就可以跳过了。但是,这种方法毕竟不完美,如果哪天另外一个目录不存在,仍然会出现这个问题。而且,logrotate 对这种场景的处理本来就不合理,然后去翻了下它的 changelog,发现这个 bug 已经在 3.7.4-12 以后的版本中 fixed:

* Thu Mar 31 2011 Jan Kaluza <jkaluza@redhat.com> - 3.7.4-12
- fix #540119 - fixed missingok problem with globs

所以,根本的解决方法是更新 logrotate 包。