Client writes fail with AEROSPIKE_ERR_FAIL_FORBIDDEN

Knowledge Base Client APIs
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Client reports error code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN

Problem Description

A client (can also be asrestore) reports an error during operation:

Error while storing record - code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN at src/main/aerospike/as_command.c:594

Explanation

This error indicates that the record cannot be written. The most common causes for this error are:

  • Writing to a set that is being deleted (set-delete for server versions prior to 3.14) or a truncate command has been issued with a timestamp preventing writes. (Check for set-delete or truncate in the server logs).

    Solution: Avoid writing to set when a delete operation is ongoing. If the set-delete was in force during the write operation, wait for the set-delete to complete before attempting to to write the records. Refer to Guidelines on Deleting Data for further details.

  • If a set reaches its stop_writes. See the Managing Data in a Set documentation page for details.

    Solution: If stop_writes is true, you will have to resolve either (depending on which threshold was reached) the lack of available disk % (min-avail-pct) or the memory usage (stop-writes-pct) before you can write any records. See the Understanding When a Server No Longer Accetps Writes article for details.

  • If received during scan, check if scan-threads-limit number of scans are running. For server versions prior to 4.7.0 - check if the scan-max-active configuration setting is reached.

    Solution: As a workaround, consider increasing the limit. Otherwise, reduce the number of ongoing scans and re-issue the scan. Refer to the Manage Scans documentation for further details.

  • If writing via a regular client when allow-nonxdr-writes is set to false or, for XDR traffic, when allow-xdr-writes is set to false. See fail_xdr_forbidden statistic for related information.

    Solution: Verify and update the configurations to ensure they are set per your use-case.

  • If clock_skew_stop_writes is in effect (strong-consistency enabled namespaces only, when cluster_clock_skew_ms is above the cluster_clock_skew_stop_writes_sec threshold).

    Solution: Refer to Strong-Consistency clock-skew Detection.

  • As of Aerospike Server 4.5.1, for each Available mode (AP) namespace where nsup is enabled (i.e. nsup-period not zero) writes will be suspended if cluster clock skew exceeds 40 seconds.

    Solution: Confirm if the cluster clock is skew within the specified limit.

  • If non-durable deletes are issued against a strong consistency enabled namespace while strong-consistency-allow-expunge is set to false (default).

    Solution: If deletes are ok to be non-durable, configure strong-consistency-allow-expunge to be true. Otherwise, consider Durable Deletes. Refer to Consistency page for further details.

  • If writing or updating records with a TTL while nsup-period is set to 0. When using Aerospike Server 4.9 or above, the default value of nsup-period is set to zero (disabled). This means that the sub-system within Aerospike that controls eviction and expiration (the namespace supervisor) is not running by default and the database will not accept records with an expiration set.

    Solution: The applicable solution depends on the desired behaviour of the database. If expiration and eviction are not required then when the record is written or updated do not set a TTL. This is done at an application level. If it is required to have records expiring or being evicted to free up space then enable nsup by changing nsup-period to a value other than zero.

    If the default-ttl is configured with a value other than 0 the database will not start with nsup-period equal to zero.

    There is a parameter called allow-ttl-without-nsup which allows the user to write records with a TTL even when nsup-period is set to zero. The purpose of this parameter is to allow testing of the amount of CPU used by the namespace supervisor (NSUP) sub-system. This parameter should not be used unless it is part of a formalised testing process. Enabling allow-ttl-without-nsup is not recommended since records might that might be expected to expire will not be removed from the device or memory. For additional discussion, see Namespace Data Retention Configuration.

Keywords

AEROSPIKE_ERR_FAIL_FORBIDDEN ASBACKUP CLIENT WRITE STOP_WRITES SET-DELETE NSUP-PERIOD ALLOW-TTL-WITHOUT-NSUP

Timestamp

May 2020