The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
AMC Is Now Community Supported!
Aerospike no longer supports the AMC, which has been replaced by the Aerospike Monitoring Stack. The AMC has been turned over to the community, with code hosted at aerospike-community/amc and documentation on the repo Wiki.
How to recover from AMC database damage
Problem Description
If a server running AMC crashes unexpectedly, it may damage the file based database used by AMC to store information. This could prevent AMC from starting up.
In such case, the following error will be printed in the AMC logs:
time="2019-06-16T03:00:01-04:00" level=info msg="Database path is: /opt/amc/amc.db"
time="2019-06-16T03:00:01-04:00" level=fatal msg="unexpected EOF"
time="2019-06-16T03:05:01-04:00" level=info msg="Database path is: /opt/amc/amc.db"
time="2019-06-16T03:05:01-04:00" level=fatal msg="unexpected EOF"
time="2019-06-17T03:00:01-04:00" level=info msg="Database path is: /opt/amc/amc.db"
time="2019-06-17T03:00:01-04:00" level=fatal msg="unexpected EOF"
Explanation
This error means that the AMC database has been damaged. To recover from this, certain files should be removed as detailed below.
Solution
Inside the AMC installation directory /opt/amc, the amc.db file can be located as well as a hidden temporary file, with a name similar to the below:
$ ls -a /opt/amc
...
.71e384d8029d70a336275f921ad8b1f595c99da7
...
To try and preserve as much data as possible, the first repair step is to remove the temporary file in isolation. The command to do this is:
rm /opt/amc/{tmpfilename}
In the above example, the name of the temporary file shown via ls -la should be substituted for {tmpfilename}.
After removing the file an attempt to start AMC should be made. If AMC still does not start, it will be necessary to remove the temporary file again as it is recreated on startup. The recreated temporary file will have a different name. The amc.db file itself should also be removed at this point.
rm /opt/amc/{tmpfilename}
rm /opt/amc/amc.db
Following this operation, AMC should successfully start up. Aerospike Enterprise Licensees should contact Aerospike Support for further help if necessary.
Keywords
AMC CRASH UNEXPECTED EOF CORRUPTION
Timestamp
September 2019