Client writes fail with AEROSPIKE_ERR_FAIL_FORBIDDEN


#1

Client reports error code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN

Problem Description

A client (can also be asrestore) reports an error during operation:

Error while storing record - code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN at src/main/aerospike/as_command.c:594

Explanation

This error indicates that the record cannot be written. The most common causes for this error are:

  • performing a write operation on a set during a set-delete being in operation (for versions prior to 3.14)
  • stop writes being enforced

To find out if during the AEROSPIKE_ERR_FAIL_FORBIDDEN a set-delete was being issued, one can grep the logs as follows:

$ grep set-delete /var/log/aerospike.log
Jun 07 2017 04:09:46 GMT: INFO (info): (thr_info.c:2675) Changing value of set-delete of ns mynamespace set segment-test to true
Jun 07 2017 04:09:46 GMT: INFO (info): (thr_info.c:3269) config-set command completed: params context=namespace;id=mynamespace;set=segment-test;set-delete=true;

Otherwise, you can find out if stop-writes is in operation as follows - if stop_writes is set to ‘true’:

$ grep stop_writes /var/log/aerospike.log
Jun 07 2017 03:47:47 GMT: WARNING (namespace): (namespace.c:440) {mynamespace} hwm_breached true (memory), stop_writes false, memory sz:93512507031 (34627453056 + 0) hwm:93200785408 sw:119829585920, disk sz:138565116032 hwm:400082599936
Jun 07 2017 04:05:38 GMT: WARNING (namespace): (namespace.c:440) {mynamespace} hwm_breached true (memory), stop_writes false, memory sz:94360724004 (35045140096 + 0) hwm:93200785408 sw:119829585920, disk sz:140236891392 hwm:400082599936

Solution

If stop-writes is true, you will have to resolve either (depending on which threshold was reached) the lack of available disk % (min-avail-pct) or the memory usage (stop-writes-pct) before you can write any records.

If the set-delete was in force during the write operation, wait for the set-delete to complete before attempting to to write the records.

Keywords

AEROSPIKE_ERR_FAIL_FORBIDDEN ASBACKUP CLIENT WRITE STOP_WRITES SET-DELETE

Timestamp

7/27/17