The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
Linux memory is used as buff/cache
Problem Description
On some versions of systemd, systemd-journald will receive SIGTERM signals from systemd. This results in journald restarting. When that happens, journald will not release file handles, resulting in memory used by logs never being released. The memory is reported as being used in buffers/cache by the process that is logging (asd), as opposed to journald itself.
Diagnosis
When this happens, you will see a lot of memory in buffers/cache and very little ‘available’ memory, as so:
$ free -m
total used free shared buff/cache available
Mem: 61405 33025 297 27814 28082 401
Swap: 2047 112 1935
You will also see SIGTERM messages in dmesg:
$ dmesg |grep SIGTERM
systemd-journald[5874]: Received SIGTERM from PID 1 (systemd)
You may also notice journald failing to deal with log files as such:
$ dmesg |grep 'systemd-journald' |grep 'file attributes'
systemd-journald[7119]: Failed to set file attributes: Operation not supported
systemd-journald[7119]: Failed to set file attributes: Operation not supported
systemd-journald[7119]: Failed to set file attributes: Operation not supported
systemd-journald[7119]: Failed to set file attributes: Operation not supported
This indicates the problem with journald. The reported memory is dirty cache and there is no way to force a flush.
Workaround
A simple temporary workaround to clear memory is a full system reboot. This will result in aerospike cold-start, but all buff/cache memory will be flushed and cleared.
For a full cleanup, you may want to backup and remove contents of journal logs from /var/log/journal prior to system restart. This should normally help as journald comes up clean and is less likely to loose file handles and be restarted by systemd.
Solution
A system upgrade to a stable version of systemd with this bug fixed resolves the issue completely. For that, please contact your distribution provider to find out which version has the issue resolved.
Keywords
systemd journald memory cache available
Timestamp
04/26/2019