Complications during and after set deletion (AER-4890)

Feature Discussion Deletion
,
1

I have attempted to delete a set at runtime following the instructions outlined here a couple of times, but unfortunately in both cases it didn’t go so well.

The first time, after a few seconds of a huge burst in writes, during which the application requests were all timeouted, the write TPS returned to normal. After running show sets in aql, I noticed there were still objects in the set and the set itself was still marked to be deleted (set-detete = true). The set remained in this state no matter how much time passed, even though running select * for the set returned 0 rows.

Even worse, the second time I tried this (on a different set), all the servers in the cluster failed during the same burst write period.

I wanted to ask if there is any throttling mechanism for the deletion so that it doesn’t use up all the cluster resources? Are there maybe plans for such a feature?

Another concern is that after restarting the servers on the second case, the aql select * command for any set returns Error: (11) AEROSPIKE_ERR_CLUSTER, even though other aql commands like show sets seem to work fine and the amc console reports all reads and writes as successful.

I am running aerospike v3.4.0.

2

This typically occurs if you haven’t issued the set-delete command on all nodes. The set-delete command triggers a delete on the master copy and deletes the replica copies on the remote nodes. If you didn’t run set-delete on all nodes then you will still have replica copies on each node.

Nsup handles the actual deletion of the set, you can increase nsup-delete-sleep to throttle the rate in which it delete. By default nsup-delete-sleep is set to 0.

We would expect the set-delete to finish in the time it takes for all nodes to complete 2 nsup cycles. By default nsup-period is 2 minutes so set-delete should complete in about 4 minutes. Depending on the number of objects in the namespace, nsup can take longer than nsup-period, in which case you would want to monitor the logs for lines similar to:

{NAMESPACE} Records: 102169954, 0 0-vt, 0(0) expired, 0(20485902) evicted, 0(0) set deletes, 0(0) set evicted. Evict ttls: 0,0,0.000. Waits: 0,0,0. Total time: 62139 ms

The “Total time” here indicates how long nusp is taking to complete; if it is greater than the nsup-period, the next nsup cycle will begin immediately after this one completes.

3

Actually I issued the set-delete command on all nodes at the same time, but still had objects left in the set and the set itself left marked to be deleted for more than a day. In the second case, where all the nodes failed at the same time during deletion all the objects were “resurrected” after restarting the server which I guess is expected since there was no time to actually delete them from the disks.

Is it maybe better to issue the set-delete command on each node separately and not at the same time?

Thank you for the tip, I will try it next time.

As I mentioned above, after issuing the set-delete command at the same time on all nodes, the set was left as set-delete = true for more than a day with objects still left inside (although just a fraction of the original number of objects). I manually set set-delete = false the next day, but even so when using select * I would get 0 rows.

Regarding the problem with getting Error: (11) AEROSPIKE_ERR_CLUSTER for all select * commands after the cluster failure on the second deletion attempt, it seems that it was fixed by itself over time…

1 Like
4

Regarding the deletes returning, please see: How is data deleted?

Basically our index is only persisted in shared memory, not disk. When deleting we only remove the entry from the index. If you reboot the machine or the process crashes the shared memory cannot be reused and the index is rebuilt from disk (in community edition the index is always rebuilt from disk). So deletes that were not overwritten on disk will return.

As for the records not deleting there has been some work around to detect anomalies that may cause this. We haven’t been able to reproduce these locally, if this is still happening in your environment if you can reproduce with the latest build we can walk you through using these tools.

No this shouldn’t matter.

1 Like
5

Aerospike 3.10.0 introduces durable deletes for Aerospike Enterprise. Learn more about how they work here: www.aerospike.com/docs/guide/durable_deletes.html.

6

I know this is an old thread, but to save making a new one I’m still encountering the same problem. I’ll give a complete breakdown…

We had a set which had somewhere in the region of 100m records in it (may be more, can’t remember) which we no longer use. I have tried numerous times to remove all these but there are always some “leftovers” as mentioned here, but when running SELECT * from ... etc it always returns 0 rows.

I have recently restarted the cluster, and this is the process I went through:

  1. Cluster restarted, one at a time (2 nodes)
  2. Once back up again, issued the following command: asadm -e "asinfo -v 'set-config:context=namespace;id=<namespace>;set=<set-name>;set-delete=true;'" (obviously replacing the <> values with namespace and set-name.
  3. Received an “OK” from both nodes, and the via AQL checked the set status:

Node 1:

+------------------+------------+----------------+----------+-------------------+-------------------------+-------------------+----------+------------+
| disable-eviction | ns         | set-enable-xdr | objects  | stop-writes-count | set                     | memory_data_bytes | deleting | tombstones |
+------------------+------------+----------------+----------+-------------------+-------------------------+-------------------+----------+------------+
| "false"          | "namespace" | "use-default"  | 236245   | 0                 | "set_name"         | 0                 | "true"   | 0          |
| "false"          | "namespace" | "use-default"  | 13073    | 0                 | "set_name"         | 0                 | "true"   | 0          |

Node 2:

+------------------+------------+----------------+----------+-------------------+-------------------------+-------------------+----------+------------+
| disable-eviction | ns         | set-enable-xdr | objects  | stop-writes-count | set                     | memory_data_bytes | deleting | tombstones |
+------------------+------------+----------------+----------+-------------------+-------------------------+-------------------+----------+------------+
| "false"          | "namespace" | "use-default"  | 3784     | 0                 | "set_name"         | 0                 | "true"   | 0          |
| "false"          | "namespace" | "use-default"  | 146      | 0                 | "set_name"         | 0                 | "true"   | 0          |

As you can see above, the set is still marked as deleting and yet it shows objects in there - this has been like this for 12 hours so far. It has deleted most of the objects (as said, from around 100m or so) but always leaves some remaining. Also, it’s worrying that the 2 nodes show different values - surely if there were objects remaining they would be the same number on both nodes? I have also ran the same command on some smaller data sets, just to tidy up the aerospike cluster, and have seen the same results - for example in one set, node 1 shows it as having 1 record and still marked as deleting and on the second node it is showing as 0 objects and deleting marked as false.

Anyone able to shed some light on this (@kporter) ?

Thanks in advance

7

Verify in your logs that there have been at least 2 nsup cycles complete since issuing the set-delete command.

The lines to look for will be similar to this:

{ns-name} Records: 638101, 0 0-vt, 922(576066) expired, 259985(24000935) evicted, 0(0) set deletes. Evict ttl: 0. Waits: 0,0,0. Total time: 155 ms
8

Yep there have been - these are a sample of the log entries…

Mar 03 2017 09:30:21 GMT: INFO (nsup): (thr_nsup.c:1097) {temp01} Records: 10968448, 3354 0-vt, 20328(19731350) expired, 0(0) evicted, 0(0) set deletes. Evict ttl: 0. Waits: 0,0,101. Total time: 1621 ms
Mar 03 2017 09:30:23 GMT: INFO (nsup): (thr_nsup.c:1097) {adspruce} Records: 1829, 1829 0-vt, 0(0) expired, 0(0) evicted, 0(57935774) set deletes. Evict ttl: 0. Waits: 0,1501,0. Total time: 1591 ms
Mar 03 2017 09:32:21 GMT: INFO (nsup): (thr_nsup.c:1097) {temp01} Records: 10968952, 3354 0-vt, 21149(19752499) expired, 0(0) evicted, 0(0) set deletes. Evict ttl: 0. Waits: 0,0,157. Total time: 1755 ms
Mar 03 2017 09:32:23 GMT: INFO (nsup): (thr_nsup.c:1097) {adspruce} Records: 1829, 1829 0-vt, 0(0) expired, 0(0) evicted, 0(57935774) set deletes. Evict ttl: 0. Waits: 0,1495,0. Total time: 1584 ms
Mar 03 2017 09:34:22 GMT: INFO (nsup): (thr_nsup.c:1097) {temp01} Records: 10969887, 3354 0-vt, 20696(19773195) expired, 0(0) evicted, 0(0) set deletes. Evict ttl: 0. Waits: 0,0,129. Total time: 1680 ms
Mar 03 2017 09:34:23 GMT: INFO (nsup): (thr_nsup.c:1097) {adspruce} Records: 1829, 1829 0-vt, 0(0) expired, 0(0) evicted, 0(57935774) set deletes. Evict ttl: 0. Waits: 0,1502,0. Total time: 1591 ms

again, currently there are still 3 sets with the set-delete true and yet they have records in and are not deleting

9

There is an undocumented setting that can be used in some of those cases, where the records not being deleted are actually prole records (there can be different issues leading to this, one of them being deletes failing to be processed on the prole side). This setting will cause a node to not only delete master records in the set that has the set-delete flag set to true, but will also delete the non-master records.

So for the nodes that have the set-delete flag stuck, this will hopefully clear those records (this may not work on old versions of Aerospike, though, I don’t remember when it was introduced from the top of my head):

asinfo -v "set-config:context=service;non-master-sets-delete=true"

Then check in the logs on all nodes that all the sets have completed 1 nsup cycle.

You can then turn this off:

asinfo -v "set-config:context=service;non-master-sets-delete=false"
10

Also there are some cases where LDT objects (deprecated) can get stuck, requiring a restart.

11

I’ve tried the above, and left it for now, but so far no change :confused:

12

Aerospike 3.12.0 was released recently and features truncate, set-delete’s successor.

Truncate was designed to be used by either operators or an applications. When the command is applied to the server, client requests for truncated records will immediately yield “not found” rather than having to wait for the individual records delete transaction to be processed. The command is more efficient since the only information that needs to be distributed is the truncate command itself, as opposed to the set-delete issuing a delete command for each record on each replica.

13

Okay, so I tried the above again and this time it seems to have sorted out the issues - thank you.

I did notice the new server version release and we’re going to update tomorrow - hopefully the truncate command will help with keeping redundant data out of the way, and do you know if it also addresses the open connections issue? Basically, aerospike thinks it has over 5k connection to each of our webheads running php-fpm, but if we check on the webheads, they think they have less than 100 which is what we would expect. Is this a known issues and something which is being corrected in a new server / client version @kporter? I can easily reproduce the issue and show you numbers / commands if that helps. Even after restarting php-fpm the aerospike server thinks the connections are still present.

I’ll open a new thread about it and tag it in here

14

I’m not familiar with the connections issue you describe, but seems @rbotzer is and has responded in the thread you opened.