Error code 4 while restoring from a backup file


#1

Error code 4 while restoring from a backup file.

Problem Description

When restoring from a backup file the following error is displayed.

2016-04-26 22:25:40 GMT [ERR] [26701] Error while storing record - code 4:
AEROSPIKE_ERR_REQUEST_INVALID at src/main/aerospike/as_command.c:545
2016-04-26 22:25:40 GMT [ERR] [26699] Error while storing record - code 4:
AEROSPIKE_ERR_REQUEST_INVALID at src/main/aerospike/as_command.c:545
2016-04-26 22:25:40 GMT [ERR] [26697] Error while storing record - code 4:

Restore does not finish and the process aborts.

Explanation

This error indicates that the server has received an invalid request. Invalid requests should not normally be generated during the restore process. One scenario where invalid requests could be sent is when there are issues with the TTL of the records being restored.

A namespace can be configured to restrict the TTL of the records it contains using the max-ttl parameter. This is a namespace level parameter that can be changed dynamically using asinfo or in a static manner by including it within the namespace stanza in aerospike.conf. By default max-ttl is set to 315360000 (10 years).

Consider a scenario where you have the max-ttl as zero and you take a backup. Let’s assume the ttl of records is 10 years. After taking the backup the max-ttl value is changed to 2 Years.

When the backup process attempts to restore records with TTL greater than the specified max-ttl the request is considered invalid and the restore process fails.

Versions 3.8.3 and above

As of version 3.8.3, the max-ttl value has been set to a default of 10 years (315360000) and can no longer be set to a value of zero. To resolve the issue, max-ttl should be set to a value higher than that of the records and up to a maximum value of 10 years.

Another solution is to update the scan and update records TTLs to a value that is less than 10 years prior to taking a backup of the cluster.

Versions prior to 3.8.3

To resolve this issue max-ttl should be set to a value higher than that of the records contained within the backup file or to zero (no restrictions). This can be done without a cluster or node restart using the following command:

asadm -e "asinfo -v 'set-config:context=namespace;id=ns1;max-ttl=0'"

Notes:

To validate this, one can also look at the server logs when the restore fails. The below lines will be printed.

Apr 27 2016 04:52:10 GMT: INFO (rw): (thr_rw.c::3345) write_local: incoming ttl 172653670 too big compared to 63072000
Apr 27 2016 04:52:10 GMT: INFO (rw): (thr_rw.c::3345) write_local: incoming ttl 172591691 too big compared to 63072000
Apr 27 2016 04:52:10 GMT: INFO (rw): (thr_rw.c::3345) write_local: incoming ttl 172659058 too big compared to 63072000
Apr 27 2016 04:52:10 GMT: INFO (rw): (thr_rw.c::3345) write_local: incoming ttl 172578653 too big compared to 63072000
Apr 27 2016 04:52:10 GMT: INFO (rw): (thr_rw.c::3345) write_local: incoming ttl 172641532 too big compared to 63072000

Details on max-ttl http://www.aerospike.com/docs/reference/configuration/#max-ttl

Keywords

BACKUP RESTORE ERROR 4 INVALID TTL

Timestamp

4/28/16