Aerospike logs rotate but are not written to when using Log Rotate to manage logs

Knowledge Base Tools
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Aerospike logs rotate but are not written to when using Log Rotate to manage logs

Problem Description

Log rotate has been installed to manage Aerospike logs. Looking at the log directory it is clear that the logs are being rotated however the rotated logs are not written to and the original log continues to grow in size.

Rotated logs will have 0 bytes file size.

-rw------- 1 root root 0 Jan 11 03:40 aerospike.log
-rw------- 1 root root 120437091486 Jan 11 08:19 aerospike.log-20151201
-rw------- 1 root root 0 Dec 1 03:17 aerospike.log-20151202
-rw------- 1 root root 0 Dec 2 03:46 aerospike.log-20151203
-rw------- 1 root root 0 Dec 3 03:33 aerospike.log-20151204
-rw------- 1 root root 0 Dec 4 03:36 aerospike.log-20151205
-rw------- 1 root root 0 Dec 5 03:43 aerospike.log-20151206
-rw------- 1 root root 0 Dec 6 03:45 aerospike.log-20151207
-rw------- 1 root root 0 Dec 7 03:11 aerospike.log-20151208
-rw------- 1 root root 0 Dec 8 03:26 aerospike.log-20151209
-rw------- 1 root root 0 Dec 9 03:09 aerospike.log-20151210
-rw------- 1 root root 0 Dec 10 03:26 aerospike.log-20151211

Root Cause

This issue will occur when the pidfile parameter is different between /etc/aerospike/aerospike.conf and /etc/logrotate.d/aerospike

The net effect of this is that Log Rotate will cycle log files but when it tries to send an HUP to the Aerospike process (asd) the signal will not be received as it does not have the right process file definition. The Aerospike process will not cycle file descriptors and continue to write to the original file descriptor which will be an old rotated log.

Solution

Edit /etc/logrotate.d/aerospike such that the pidfile mentioned in the following command matches that held in aerospike.conf

/bin/kill -HUP `cat /var/run/aerospike/asd.pid`

For systemd system, asd.pid is not generated, then use this instead:

/bin/kill -HUP `pgrep asd`

Then restart Log Rotate to pick up the changes. It may be necessary to do manual housekeeping on the existing logs

Notes

  1. You can use “logrotate -f -d” to debug logrotate related issues. See the man page of logroate for details.
  2. The “-f” will force the rotation, even if it doesn’t think this is necessary.
  3. You may also use “copytruncate” if you specify more than one log file. See this link for details: Post Aerospike 3.8 XDR logs do not rotate
  4. If the filesystem is full and logrotate fails, you can use these commands to backup the last 100K lines of aerospike.log and then reduce the aerospike.log to 100MB in size:
mkdir -p /tmp/aerospike
mount -o size=150M -t tmpfs none /tmp/aerospike
sudo tail -n 100000 aerospike.log > /tmp/aerospike/aerospike.log
sudo truncate --size=100M aerospike.log

Keywords

LOGROTATE FULL ROTATE

Timestamp

5/15/2017

1 Like