How to recover from AMC database damage

How to recover from AMC database damage

Problem Description

If a server running AMC crashes unexpectedly, it may damage the file based database used by AMC to store information. This could prevent AMC from starting up.

In such case, the following error will be printed in the AMC logs:

time="2019-06-16T03:00:01-04:00" level=info msg="Database path is: /opt/amc/amc.db"
time="2019-06-16T03:00:01-04:00" level=fatal msg="unexpected EOF"
time="2019-06-16T03:05:01-04:00" level=info msg="Database path is: /opt/amc/amc.db"
time="2019-06-16T03:05:01-04:00" level=fatal msg="unexpected EOF"
time="2019-06-17T03:00:01-04:00" level=info msg="Database path is: /opt/amc/amc.db"
time="2019-06-17T03:00:01-04:00" level=fatal msg="unexpected EOF"

Explanation

This error means that the AMC database has been damaged. To recover from this, certain files should be removed as detailed below.

Solution

Inside the AMC installation directory /opt/amc, the amc.db file can be located as well as a hidden temporary file, with a name similar to the below:

$ ls -a /opt/amc
...
.71e384d8029d70a336275f921ad8b1f595c99da7
...

To try and preserve as much data as possible, the first repair step is to remove the temporary file in isolation. The command to do this is:

rm /opt/amc/{tmpfilename}

In the above example, the name of the temporary file shown via ls -la should be substituted for {tmpfilename}.

After removing the file an attempt to start AMC should be made. If AMC still does not start, it will be necessary to remove the temporary file again as it is recreated on startup. The recreated temporary file will have a different name. The amc.db file itself should also be removed at this point.

rm /opt/amc/{tmpfilename}
rm /opt/amc/amc.db

Following this operation, AMC should successfully start up. Aerospike Enterprise Licensees should contact Aerospike Support for further help if necessary.

Keywords

AMC CRASH UNEXPECTED EOF CORRUPTION

Timestamp

September 2019

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.