The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
FAQ On RHEL system abrt states that the running binary asd has been deleted
Detail
On a RHEL system, abrt is a standard process for monitoring binaries that are on disk and in memory. Occassionally abrt may report rather concerning messages about the asd binary being removed, as follows:
Jan 24 02:11:48 aerospike001 abrt[30605]: File '/usr/bin/asd' seems to be deleted
Jan 24 02:11:54 aerospike001 abrt[30605]: Saved core dump of pid 22143 (/usr/bin/asd) to /var/spool/abrt/ccpp-2018-01-24-02:11:48-22143 (1048576000 bytes)
Jan 24 02:11:54 aerospike001 abrtd: Directory 'ccpp-2018-01-24-02:11:48-22143' creation detected
Jan 24 02:11:55 aerospike001 abrtd: Package 'aerospike-server-enterprise' isn't signed with proper key
Jan 24 02:11:55 aerospike001 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2018-01-24-02:11:48-22143' exited with 1
Jan 24 02:11:55 aerospike001 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2018-01-24-02:11:48-22143'
Answer
The first line indicates the file has been deleted, but that’s not what abrt means. What this line is actually stating is that the binary on disk, /usr/bin/asd was loaded and running in RAM and while the binary is still running, the file from which it was started has changed.
This may be caused by a number of things, but simply changing the update/modify time of the binary or running an upgrade of aerospike while the asd is still running will result in this error.
Typically, when this happens, abrt will terminate the asd binary immediately, resulting in it being stopped and not generating any logs (abrubtly stopping logging).
ABRT will crteate a directory, as state in its logs, in /var/spool/abrt/… and put the core dump in there. If you have enabled unlimited core dumps, there may not be enough space on the root disk to store the core dump, and, as is evident from the links above, this will result in abrt deleting the core dump directoy. Therefore no core dumps will be stored.
We can see from the above log, that abrt complains about aerospike-server-enterprise not being signed by a key that RHEL would have imported. This will tell us that abrt noticed that there was an upgrade run of the package as well. As the timestamps are near, we can only deduct that the aerospike package was being installed while asd was running, resulting in this issue.
The solution to this is to ensure that asd is indeed stopped before running the upgrade.
Issuing a stat will explain what changed:
# stat /usr/bin/asd
File: `/usr/bin/asd'
Size: 52835482 Blocks: 103200 IO Block: 4096 regular file
Device: 803h/2051d Inode: 11142903 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-01-24 02:26:34.734579604 -0600
Modify: 2017-06-23 21:28:19.000000000 -0500
Change: 2018-01-06 03:37:48.430082966 -0600
Notes
ABRT will typically report this even if modify or update timestamp on the file changed. Issuing a simple ‘touch /usr/bin/asd’ will result in abrt complaining already.
ABRT may report things incorrectly / with a delay, if its database is corrupted. It is recommended to not install it if you don’t absolutely need it, as it has been known to produce false-positive alerts for old changes.
ABRT issuing a false-positive may also be indicative or memory/RAM corruption. If the file hasn’t changed, it is advisable to test your RAM.
Keywords
ABRT ASD DELETED
Timestamp
2/1/18