Aerospike log files consume space on file system despite deletion

Knowledge Base Operating System
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Aerospike log files consume space on file system despite deletion

Problem Description

The aerospike.log file is deleted to free space on a file system but the space is not released and when the lsof command is used to see what files are open, output similar to the following is produced.

asd        3004         root    4w   REG    8,8 3120790837   78 /var/log/aerospike/aerospike.log-20190515 (deleted)

Explanation

This issue will occur when the aerospike.log file has been deleted while it is still in use by the Aerospike process (asd). When a file is opened the process that opens the file holds a file descriptor which is used to write to the file itself. The file will only be deleted when there are no processes with open file descriptors pointing to it. Storage space will not be reclaimed when an open file is deleted. When there are no further open file descriptors the operating system will reclaim the space.

Solution

If the log file has been deleted and the Aerospike service has not been restarted, then it may be possible to recover the log data using the following steps.

  • Check lsof to see if the asd process is still holding onto the file descriptor…
  • Run the following command

$ tail /proc/<pidof asd>/fd/*

This will show which file descriptor the asd process is writing to.

  • Using the following file descriptor 5 and the process ID of the asd is 7539 there will be a file called /proc/7539/fd/5. which represents the deleted file with the log data.
  • Copy the file to a backup directory.

$ cp /proc/7539/fd/5 /mnt/backup/aerospike.log.backup

  • Issue a SIGHUP signal to the asd process which will roll the log and start logging to a new file.

$ kill -HUP 7539

If it is not important to retain the log data the following command can be used to roll the logs to a new file. This will close the file descriptor and allow space to be reclaimed.

$ kill -HUP 7539

Notes

  • There is an automatic way to manage Aerospike log rotation with the Logrotate tool.

Keywords

LSOF LOG DELETED SPACE

Timestamp

August 2019