Linux memory is used as buff/cache

#1

Linux memory is used as buff/cache

Problem Description

On some versions of systemd, systemd-journald will receive SIGTERM signals from systemd. This results in journald restarting. When that happens, journald will not release file handles, resulting in memory used by logs never being released. The memory is reported as being used in buffers/cache by the process that is logging (asd), as opposed to journald itself.

Diagnosis

When this happens, you will see a lot of memory in buffers/cache and very little ‘available’ memory, as so:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:          61405       33025         297       27814       28082         401
Swap:          2047         112        1935

You will also see SIGTERM messages in dmesg:

$ dmesg |grep SIGTERM

systemd-journald[5874]: Received SIGTERM from PID 1 (systemd)

You may also notice journald failing to deal with log files as such:

$ dmesg |grep 'systemd-journald' |grep 'file attributes'

systemd-journald[7119]: Failed to set file attributes: Operation not supported
systemd-journald[7119]: Failed to set file attributes: Operation not supported
systemd-journald[7119]: Failed to set file attributes: Operation not supported
systemd-journald[7119]: Failed to set file attributes: Operation not supported

This indicates the problem with journald. The reported memory is dirty cache and there is no way to force a flush.

Workaround

A simple temporary workaround to clear memory is a full system reboot. This will result in aerospike cold-start, but all buff/cache memory will be flushed and cleared.

For a full cleanup, you may want to backup and remove contents of journal logs from /var/log/journal prior to system restart. This should normally help as journald comes up clean and is less likely to loose file handles and be restarted by systemd.

Solution

A system upgrade to a stable version of systemd with this bug fixed resolves the issue completely. For that, please contact your distribution provider to find out which version has the issue resolved.

Keywords

systemd journald memory cache available

Timestamp

04/26/2019