The asd process cold starts after a dirty exit, which appears to be clean

The asd process cold starts after a dirty exit, which appears to be clean

Problem Description

When starting asd after a successful process shutdown, with clear logs, asd always cold starts for no apparent reason.

Note that this only applies for Enterprise Edition as Community Edition will always cold start. Please also note that there are cases where asd cold starts after a graceful shutdown is normal. Such cases include:

  • shared memory cleared manually, or server rebooted
  • aerospike manually cold started
  • a disk (or disks) has been cleared and is empty
  • when using data-in-memory true on Aerospike versions below 3.15.1.3

Explanation

Typically, once Aerospike is stopped, and the last log line is printed, the asd process will still run for a short while in order to clear the memory. The last log line in question for successful shutdown is:

Jul 01 2019 08:16:54 GMT: INFO (as): (as.c:449) finished clean shutdown - exiting

In some cases, the shutdown process can take longer.

  • When a large number of sprigs have been configured, this process may take a while as sprigs root have to be re-written in shared memory.

  • When using pmem index-type, as the CPU cache has to be persisted on shutdown.

Unfortunately, the systemd service has a default timeout for how long to allow a process to exit after issuing SIGTERM, this is set to 15 seconds. After this timeout, systemd will issue a SIGKILL.

Solution

If this is the case, the systemd configuration can be changed to allow more time for the process to exit cleanly.

The /usr/lib/systemd/system/aerospike.service should be edited to add/edit the following line under [Service] heading. This configures the timeout prior to the SIGKILL, the value in the example below sets this to 900 seconds.

TimeoutSec=900

Notes

  • The example shown above uses 900 seconds as a timeout value. In reality this is excessively high. To determine the correct value an initially high value should be chosen, the process should be stopped and the pidof asd command should be used to check the correct time required to shutdown the asd process.

  • Documentation on cold start.

  • Version 4.6 changes this default setting to 10 minutes.

Keywords

ASD SYSTEMD COLDSTART

Timestamp

June 2019

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.