Aerospike log files consume space on file system despite deletion
The aerospike.log file is deleted to free space on a file system but the space is not released and when the
lsof command is used to see what files are open, output similar to the following is produced.
asd 3004 root 4w REG 8,8 3120790837 78 /var/log/aerospike/aerospike.log-20190515 (deleted)
This issue will occur when the aerospike.log file has been deleted while it is still in use by the Aerospike process (
asd). When a file is opened the process that opens the file holds a file descriptor which is used to write to the file itself. The file will only be deleted when there are no processes with open file descriptors pointing to it. Storage space will not be reclaimed when an open file is deleted. When there are no further open file descriptors the operating system will reclaim the space.
If the log file has been deleted and the Aerospike service has not been restarted, then it may be possible to recover the log data using the following steps.
lsofto see if the asd process is still holding onto the file descriptor…
- Run the following command
$ tail /proc/<pidof asd>/fd/*
This will show which file descriptor the
asd process is writing to.
- Using the following file descriptor 5 and the process ID of the
asdis 7539 there will be a file called /proc/7539/fd/5. which represents the deleted file with the log data.
- Copy the file to a backup directory.
$ cp /proc/7539/fd/5 /mnt/backup/aerospike.log.backup
- Issue a SIGHUP signal to the
asdprocess which will roll the log and start logging to a new file.
$ kill -HUP 7539
If it is not important to retain the log data the following command can be used to roll the logs to a new file. This will close the file descriptor and allow space to be reclaimed.
$ kill -HUP 7539
- There is an automatic way to manage Aerospike log rotation with the Logrotate tool.
LSOF LOG DELETED SPACE