Enabling Coredumps on RedHat/CentOS

Knowledge Base Deploying
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Synopsis

Customer may need to enable Core dumps to help investigate stack traces.

Non-systemd environment:

To enable core dumps for all daemon, please follow these steps: Edit the /etc/profile. At line 26 of the file, replace this line:

ulimit -S -c 0 > /dev/null 2>&1
with this line:
ulimit -c unlimited >/dev/null 2>&1

Replace this line (around line 138 on Fedora) in /etc/init.d/functions

ulimit -S -c 0 >/dev/null 2>&1
with this:
ulimit -S -c ${DAEMON_COREFILE_LIMIT:-0} >/dev/null 2>&1

Enable core-dumping globally by editing the /etc/sysconfig/init file and adding the line:

DAEMON_COREFILE_LIMIT='unlimited'

Enable it for specific daemons by adding this line in the /etc/sysconfig/$daemon:

DAEMON_COREFILE_LIMIT='unlimited'

Optionally, enable core dump for SUID programs:

echo 1 > /proc/sys/fs/suid_dumpable
Edit the /etc/sysctl.conf and add the following:
fs.suid_dumpable = 1
kernel.core_uses_pid = 1
Then reload the settings in /etc/sysctl.conf:
sysctl -p

systemd environments

  • Create a location for the Core files

    sudo mkdir -p /opt/corefiles/

chmod 0777 /opt/corefiles

  • Set kernel.core_pattern to desired path for coredump files

    • Dynamically through sysctl :
sysctl -w kernel.core_pattern=/opt/corefiles/core.%e.%t.%p
sysctl -w kernel.core_pipe_limit=0
sysctl -w kernel.core_uses_pid=0
sysctl -w fs.suid_dumpable=1

  • Or make the changes persistent in /etc/sysctl.conf and run sudo sysctl -p

  • Verify Core pattern

    • cat /proc/sys/kernel/core_pattern

  • Edit /etc/security/limits.conf

    • add the following line entry at the bottom of the file (before # End of file)
    *       hard        core        unlimited
*       soft        core        unlimited

exit and re-enter current shell.

  • Modify /etc/systemd/system.conf
    • add DefaultLimitCORE=infinity
  • Modify your systemd aerospike service configuration
    • vi /etc/systemd/system/aerospike.service.d/override.conf
    • add the following:
[Service]
LimitCORE=infinity
  • reload the new systemd config and restart aerospike service
sudo systemctl daemon-reexec
sudo systemctl stop aerospike
sudo systemctl start aerospike
sudo systemctl status aerospike

Test coredump

  • Force asd to do a SIGSEGV
 sudo kill -11 `pidof asd`
  • Generate a kernel panic in non-prod environment
echo c > /proc/sysrq-trigger