Configuring rsyslogd to avoid aerospike hang with security enabled


#1

Configuring rsyslogd to avoid aerospike hang with security enabled

Problem description

When configuring ‘security’ in aerospike, you have a choice of logging either to log file or to syslog. When logging to syslog, you may notice aerospike hanging until rsyslog is restarted.

Explanation

rsyslog uses an internal in-memory buffer for writes. If this buffer is full, rsyslog will stop accepting connections. This isn’t a problem if you are using UDP, as the protocol will discard packets in that case. If you use local socket logger or TCP, though, this will create a lock on the applications which, by design, must log to continue.

As such, if you, for example, use rsyslog on the local aerospike node to forward logs to a central location using TCP, should the central location become unavailable/hung (or network-overloaded), local rsyslog will very quickly get to this hung state causing aerospike to stop responding.

Solution

There are 2 available solutions with rsyslog. The first one should be implemented either way, with second being optional to further ensure aerospike won’t hang because of rsyslog.

rsyslog buffers

rsyslog allows for the configuration of on-disk buffers, further extending the in-memory buffers. Configuring those is highly recommended to allow for a greater survival rate of rsyslog being backed up. This will also help persist hung messages throughout restarts, further ensuring logs don’t get lost.

This is the example configuration for shipping syslogs to a remote desitnation using TCP rsyslog with disk-backed buffers (added to /etc/rsyslog.conf)

# action forwarded with 1GB set aside for the message queue
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName fwdqueue1 # set file name, also enables disk mode. will be stored in /var/spool/rsyslog
# Set ActionQueueSpoolDirectory if you want to redirect away from /var/spool/rsyslog
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionQueueMaxFileSize 1024M # how much memory to allow for the queue size max
*.* @@10.0.0.1:514 # where to forward. @@ means TCP. Replace with IP:PORT of remote machine

rsyslog file input pickup

rsyslog also allows to pickup logs from files, instead of socket/udp/tcp/relp. This allows you to configure and use aerospike with local logging and to use rsyslog to pickup those logs and ship them. This ensures aerospike hang will not happen, ever. This should be used in addition to the above solution.

Example imfile log pickup (in /etc/rsyslog.conf)

module(load="imfile")
$InputFileName /var/log/aerospike.log
$InputFileTag aerospike-node1
$InputFileStateFile aerospike-state
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFilePersistStateInterval 1000

Notes

Example aerospike configuration for file logging in info and for security:

logging {
    file /var/log/aerospike.log {
        context any info
    }
}
security {
    enable-security true
    # Write the audit trail to log
    log {
        report-authentication true
        report-user-admin true
        report-sys-admin true
        report-violation true
    }
}

Example security context logging to syslog:

security {
    enable-security true
    # Write the audit trail to syslog
    syslog {
        local 0 # write to "local0" facility as well as to default syslog sink
        report-authentication true
        report-user-admin true
        report-sys-admin true
        report-violation true
    }
}

Keywords

RSYSLOG SYSLOG HANG ASD SECURITY

Timestamp

06/05/2018