Linux memory is used as buff/cache
On some versions of systemd, systemd-journald will receive SIGTERM signals from systemd. This results in journald restarting. When that happens, journald will not release file handles, resulting in memory used by logs never being released. The memory is reported as being used in buffers/cache by the process that is logging (asd), as opposed to journald itself.
When this happens, you will see a lot of memory in buffers/cache and very little ‘available’ memory, as so:
$ free -m total used free shared buff/cache available Mem: 61405 33025 297 27814 28082 401 Swap: 2047 112 1935
You will also see SIGTERM messages in dmesg:
$ dmesg |grep SIGTERM systemd-journald: Received SIGTERM from PID 1 (systemd)
You may also notice journald failing to deal with log files as such:
$ dmesg |grep 'systemd-journald' |grep 'file attributes' systemd-journald: Failed to set file attributes: Operation not supported systemd-journald: Failed to set file attributes: Operation not supported systemd-journald: Failed to set file attributes: Operation not supported systemd-journald: Failed to set file attributes: Operation not supported
This indicates the problem with journald. The reported memory is dirty cache and there is no way to force a flush.
A simple temporary workaround to clear memory is a full system reboot. This will result in aerospike cold-start, but all buff/cache memory will be flushed and cleared.
For a full cleanup, you may want to backup and remove contents of journal logs from
/var/log/journal prior to system restart. This should normally help as journald comes up clean and is less likely to loose file handles and be restarted by systemd.
A system upgrade to a stable version of systemd with this bug fixed resolves the issue completely. For that, please contact your distribution provider to find out which version has the issue resolved.
systemd journald memory cache available