How to clear up set and bin names when it exceeds the maximum set limit


#1

FAQ How to clear up set and bin names when it exceeds the maximum set limit.

Please note that the set-delete command has been deprecated and the truncate command should now be used.

  • truncate introduced in Aerospike Server version 3.12.0, released in March 2017.
  • set-delete deprecated and works up to Aerospike 3.12.1, released in April 2017.

The version of the Aerospike Server being maintained will determine which of the following asinfo commands to execute and whether to follow the set-delete instructions listed below or use truncate.

Context

Deleting set and bin names doesn’t delete the associated meta data. At some point the maximum limit for sets and bins would be breached. The following log entries would show up, indicating no more bin or set names can be added.

WARNING (bin): (bin.c::377) {test-namespace} bin-name quota full - can't add new bin-name
OR
WARNING (namespace): (namespace.c:580) at set names limit, can't add set-name

Method (truncate, post version 3.12)

Execute the truncate command for the set (or the namespace to clear up bin names across a namespace):

asinfo -v "truncate:namespace=<namespace name>[;set=<set name>]"

Issue a cold restart on each node in a rolling fashion (waiting for migrations to complete before taking the next node down depends on the version, as the new cluster protocol introduced in version 3.13/3.14 does not require to wait for migrations to complete prior to taking the next node down).

Be considerate of records of other sets that may be brought back as well. Deleting the whole persistent storage will avoid other deleted records, from other sets to come back. This would then require waiting for migrations to complete (to repopulate records on the empty namespace) prior to taking the next node down.

Method (set-delete, prior to version 3.12)

A clean up of the persisted storage for the relevant namespace, and accompanying user roles, followed by a start up will fully remove the set/bin names. But this would only work if the replica nodes have no references of the deleted sets or bins. It is important to do it for one node at a time, in a rolling fashion. The commands listed below refer to a set but the same steps can be used for bin names as well (replace with a bin name where applicable).

Step 1:

Run the set delete command for every node in the cluster (or run it from asadm to hit all the nodes):

asinfo -v "set-config:context=namespace;id=<name space>;set=<set name>;set-delete=true;"
Step 2:

Ensure that all records in the set are deleted (n_objects is 0) and set-delete is back to “false”, for example, using aql:

aql> show sets
+-----------+------------------+----------------+-------------------+----------------+--------------+------------+------------+
| n_objects | disable-eviction | set-enable-xdr | stop-writes-count | n-bytes-memory | ns_name      | set_name   | set-delete |
+-----------+------------------+----------------+-------------------+----------------+--------------+------------+------------+
| 0         | "false"          | "use-default"  | 0                 | 0              | "test_file1" | "testset1" | "false"    |
+-----------+------------------+----------------+-------------------+----------------+--------------+------------+------------+
Step 3:

Verify that no user roles (“read-write-ns-set”) with privileges on the set in that namespace (“read-write.test_file1.testset1”) exist, for example, using aql:

aql> show roles
+---------------------+----------------------------------+
| role                | privileges                       |
+---------------------+----------------------------------+
| "data-admin"        | "data-admin"                     |
| "read-write-ns-set" | "read-write.test_file1.testset1" |
| "read"              | "read"                           |
| "read-write"        | "read-write"                     |
| "read-write-udf"    | "read-write-udf"                 |
| "sys-admin"         | "sys-admin"                      |
| "user-admin"        | "user-admin"                     |
+---------------------+----------------------------------+
Step 4:

Drop any user roles(“read-write-ns-set”) with privileges on the set in that namespace(“read-write.test_file1.testset1”), for example, using aql:

aql> drop role <role name>
Step 5:

Verify that no user roles (“read-write-ns-set”) with privileges on the set in that namespace (“read-write.test_file1.testset1”) exist, for example, using aql:

aql> show roles
+------------------+------------------+
| role             | privileges       |
+------------------+------------------+
| "data-admin"     | "data-admin"     |
| "read"           | "read"           |
| "read-write"     | "read-write"     |
| "read-write-udf" | "read-write-udf" |
| "sys-admin"      | "sys-admin"      |
| "user-admin"     | "user-admin"     |
+------------------+------------------+

Steps 6 & 7 should be repeated for each node in the cluster.

Step 6:

Stop the aerospike daemon on a node then clear the data from the persistent store.This step is only applicable when the replication factor is 2 or more on the namespace

For SDD, run the dd command for the device which is attached to the particular namespace and set:

sudo dd if=/dev/zero of=/dev/<deviceID> bs=1M&

OR For rotational drives, remove the file associated with the particular namespace and set:

sudo rm /opt/aerospike/<filename>
Step 7:

Start the aerospike daemon.

Depending on the specifics of the workload and potential other namespaces in the cluster, you may need to wait for migrations to complete before moving on to the next node.

Step 8:

Repeat Step 6 & 7 for every node in the cluster

Keywords

BIN SET LIMIT ROLE DELETE TRUNCATE

Timestamp

12 June 2017


Cannot drop sets in namespaces