Guidelines for Deleting Data

Knowledge Base General
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Details

Please note that the set-delete command has been deprecated. The truncate command should be used instead.

  • The truncate asinfo command was introduced in Aerospike Server version 3.12.0, released in March 2017. (Records truncated will not be restored upon cold restart when using the Enterprise Edition).

  • set-delete asinfo command deprecated and works up to Aerospike 3.12.1, released in April 2017.

  • Scan and Delete. Supported programmatically by all Clients.

  • Clients can also of course issue deletes or durable-delete (introduced in Aerospike Server version 3.10.0.3) for deleting specific records. The relevant policy needs to be set prior to issuing a delete.

The version of the Aerospike Server being maintained will determine which of the following asinfo commands to execute and whether to follow the set-delete instructions listed below or use truncate.

Truncate (Aerospike Server version 3.12.1 and above)

Test Truncations Gradually

It is recommended to test the truncations gradually, one set at a time and monitor the potential impact on the overall system performance. Even though truncation is much more efficient than its predecessor set-delete, (for example it does not generate deletes that propagate to the replicas over fabric), the vacuum created by the sudden deletion of records could, for example, cause a surge in defragmentation activity, impacting the storage subsystem performance. This is specifically important in situations where defragmentation settings have been changed from their default or when having a large number of partitions on top of the underlying physical storage device.

Note: Monitoring on the Client Side

  • Please note that from the user (application) side, truncates are effectively instantaneous
  • There is no need to monitor the truncate progress from the application point of view of discovering when truncates have completed
  • This article discusses monitoring truncate progress from the POV of server administration

Usage

Truncate a set

asinfo -v "truncate:namespace=namespace_name;set=set_name"

Use truncate command. The name of the set to be truncated would have to be specified.

WARNING: For some older Aerospike Server versions, prior to 4.6, not specifying the set would truncate the whole namespace.

Truncate a set up to a specific time

asinfo -v "truncate:namespace=namespace_name;set=set_name;lut="

The last updated time is expressed in nanoseconds (8 bytes) since the UNIX epoch (i.e., nanoseconds since 00:00:00 UTC on 1 Jan 1970). It can be given in hex (with a 0x prefix), decimal, or octal (with a 0 prefix). The lut time is not allowed to be older than the Citrusleaf epoch (00:00:00 UTC on 1 Jan 2010), and is not allowed to be beyond the current time (i.e. in the future). If not specified the current time is used.

Truncate a namespace

  • Use the truncate-namespace command, available in Aerospike Server versions 4.3.1.11 (and higher 4.3.1.X), 4.4.0.11 (and higher 4.4.0.X), 4.5.0.6 (and higher 4.5.0.X) and 4.5.1.5 and higher:
asinfo -v 'truncate-namespace:namespace=namespace_name;'
  • Use the truncate command for versions other than mentioned above:
asinfo -v "truncate:namespace=namespace_name"

Log Analysis

Here are the log lines which will provide more information on the status and progress of a truncate command. See Log Reference for further details.

The log line formats below are from version 3.12.1:

Apr 13 2017 00:41:24 GMT: INFO (truncate): (truncate.c:206) {test|testset} got command to truncate to now (229740084581)

Truncate command received.

NOTE: Will only appear on the node to which the info command was issued. The command is distributed to other nodes via system metadata (SMD), and only the truncating/starting/restarting/truncated/done log entries will appear on those nodes.

The timestamp printed in the logs is the truncation time but represented in milliseconds since the Citrusleaf epoch (00:00:00 UTC on 1 Jan 2010).

Apr 13 2017 00:41:24 GMT: INFO (truncate): (truncate.c:440) {test|testset} truncating to 229740084581

Truncate command received. Will appear on all the nodes after a truncate command is issued.

Apr 13 2017 00:41:24 GMT: INFO (truncate): (truncate.c:462) {test} starting truncate

Truncate command being processed for the namespace.

Apr 13 2017 00:41:27 GMT: INFO (truncate): (truncate.c:569) {test} truncated records (10,50)

Truncate command being processed for the namespace. The numbers in parentheses represent (current,total).

current is the number of records that have been deleted by truncation since the command was issued (10 in this example).

total is the number of records that have been deleted by truncation since the server started (50 in this example).

Counts are only kept at the namespace level.

Apr 13 2017 00:41:27 GMT: INFO (truncate): (truncate.c:573) {test} done truncate

Truncate command completed.

Highlighted Guidelines

  • If using the client APIs to perform the truncate command on a single-threaded application it is suggested to add a millisecond (ms) sleep. The truncate operation has a 1 millisecond resolution and writes occurring within the same millisecond are not deleted.
  • If using the client APIs to perform the truncate command on a multi-threaded application please be aware that accuracy of the truncate is to a precision of 1 millisecond(ms). Therefore, any writes occurring within that 1 millisecond(ms) need to be considered as those records will still persist.
  • A truncate done for a huge number of records would build up the defrag-q immediately and would impact the write-q. For server versions prior to 5.1, this could lead to memory usage spikes and even OOM’s. See article on Understanding device write-q impact on memory for further details.

FAQ

Truncate Permissions

  • If security is enabled on the cluster ( enable-security true), the user executing the truncate command must have been granted the sys-admin or data-admin role for Aerospike server versions prior to 5.1. For Aerospike server versions 5.1 and above, a user with the write permission would be able to issue the truncate command as well.

What happens in situations where there are ongoing writes on a set/namespace being truncated?

  • New writes can be performed into the sets being truncated, as the new writes will occur after the LUT the truncate command was issued with (unless the command is issued with an LUT in the future).

How fast would the truncate command progress?

  • The truncate command rapidly removes entries from the primary index. There is no dependency on the namespace supervisor (nsup) cycle thread, so deletion begins immediately upon initiation of the truncate command and will go as fast as the primary index is reduced (based on the load of the system and the number of configured truncate-threads. If secondary indices are configured, it may slow down the truncation as records would have to be read in order for their potential secondary index entries to be cleaned up. If the use cases permits it, removing the secondary indices associated with the set to be truncated (or the namespace for truncating a whole namespace) would improve the truncation speed.

Can I use the truncate command from a Client?

Can I bring back truncated records?

  • Enterprise Edition only: In consideration of the case of a cold start, an entry is added in the SMD (System Meta Data) subsystem so that a full restart does not cause the data to return. This can be considered as the “tombstone” for the set (or the namespace if truncating a whole namespace) and prevent records from coming back during a cold start. See Impact of Cold Start for more details. If there is a requirement to bring back truncated records, use truncate-undo and truncate-namespace-undo.

How do I calculate the LUT needed for the truncate command?

How does truncate handle cluster changes?

  • Truncation essentially takes effect immediately across the cluster, and will apply to any migrations that might be in progress.

  • Truncation will apply to any new nodes joining or old nodes rejoining the cluster after the truncate was executed as the truncated time would be synced over the SMD protocol.

Can I issue multiple truncate commands in parallel?

  • Yes, Aerospike can truncate 2 sets in parallel. If a truncate command is received for the same set, while another truncation is in progress, we will flag truncation for restart and issue it once the current truncate command pass is completed. See log reference for further details.

  • The truncate-threads configuration parameter (for server versions 4.6 and above) can be tuned if needed to speed up truncation.

Does truncate clear out set-metadata as well?

  • If set metadata are required to be deleted as well, any potential roles with privileges as well as secondary indices on the set in that namespace would need to be dropped prior to cold starting each node in the cluster in a rolling fashion (Enterprise Edition only). See article on Clearing out Set Names for further details.

Is there a programmatic way to ensure the truncation is completed?

  • There is no direct statistic to confirm if a truncate command has completed. There is an improvement request being considered and filed under internal Jira ticket PROD-1177. We recommend monitoring object count statistics truncated_records at the namespace level along with the messages in the Aerospike logs.

Does truncation free up space immediately?

  • See knowledge-base article on Truncation for details.

set-delete (Aerospike Server versions 3.12.0 or below)

Test Deletions Gradually

Meaning if you want to delete multiple sets of data, do not start out deleting all sets at once. Rather, delete 1 set initially and verify overall system impact. Then gradually increase the sets to be deleted monitoring the system during each iteration.

Monitor System Impact

There will be an overall system impact when deleting your data. Deletions will propagate over fabric and could impact network efficiency. Deletes can also impact the defragmentation rate, amplifying writes and potentially increasing transaction latencies.

When considering deletion you should consider tuning, as you will have to choose between:

A. Records being deleted at a quick pace, with a greater impact on latency

B. Records being deleted at a slower pace, with lesser impact on latency.

Tune Configuration Parameters

Some configuration parameters to consider to tune would be:

Log Analysis

Here are the two log lines which will provide more information on the status of set delete.

The log line formats below are from version 3.9.

{ns-name} Records: 37118670, 0 0-vt, 0(377102877) expired, 185677(145304222) evicted, 0(0) set deletes, 0(0) set evicted. Evict ttls: 34560,38880,0.118. Waits: 0,0,8743.

set deletes represents the number of records deleted by a set-delete command.

in-progress: tsvc-q 0 info-q 5 nsup-delete-q 10 rw-hash 0 proxy-hash 0 tree-gc-q 0

nsup-delete-q represents the number of records queued up for deletion by the nsup thread.

Highlighted Guidelines

Here are the most stressed highlighted guidelines:

  • If security is enabled (enable-security true), the user executing the set-delete command must have been granted the ‘sys-admin’ or ‘data-admin’ role.
  • Any roles with privileges on the set in that namespace would need to be dropped prior to truncating the set, as the metadata still exists and could return on a subsequent cold start.
  • In considering the case of a cold start, the index will be rebuilt from persistent storage and hence, deleted data for which the defragmentation has not processed or has processed but not yet overwritten will reappear.
  • In order to delete all the objects in the set in a cluster, set-delete needs to be dynamically set to true on all nodes.
    • For example: 
      asadm -e “asinfo -v ‘set-config:context=namespace;id=test;set=testset;set-delete=true;’”
  • In order to delete the set data cleanly, ensure that set-delete shows as false, on ALL nodes, after executing the deletion command.
  • Do not perform writes into the sets being deleted.
  • Do not perform set-delete during migrations as this can result in records migrated into a partition after it has been processed for the set-delete, but before the whole set is deleted, resulting in the node being stuck in the ‘set-delete’ true state.

References

Keywords

SET-DELETE DELETE DATA SET RECORDS LATENCY DEFRAGMENTATION DELETES TRUNCATE LUT NAMESPACE

Timestamp

August 2020