The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
Descriptive title of issue
Aerospike Server version 3.12.1 or above
As of version 3.12.1, released in April 2017, set-delete
is deprecated. Truncate the new feature of deleting all the data in a namespace or set is now supported in the database.
Using the truncate
command will not prolong stop-writes.
See the following truncate
documentation: Info Command Reference | Aerospike Documentation
Truncate can also be executed from the client APIs. Here is the java documentation: AerospikeClient - aerospike-client 7.2.0 javadoc
Aerospike Server version 3.12.1 or below
Problem Description
A namespace goes into stop-writes as the memory usage exceeds the stop-writes-pct threshold. Even though subsequently the memory drops below the threshold, the stop-writes remains set when the namespace is viewed in asadm
====ASCOLLECTINFO====
['namespace']
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace Node Avail% Evictions Master Replica Repl Stop Pending Disk Disk HWM Mem Mem HWM Stop
. . . . Objects Objects Factor Writes Migrates Used Used% Disk% Used Used% Mem% Writes%
. . . . . . . . (tx%,rx%) . . . . . . .
problem server03.customer.com:3000 65 0 14.261M 14.814M 2 true (0,0) 34.612GB 24 50 75.01 GB 75.01 80 75%
Initially the logs show that stop-writes is true.
Jul 26 2016 01:32:01 GMT: WARNING (namespace): (namespace.c:440) {problem} hwm_breached false, stop_writes true (memory), memory sz:80548665920 (80548665920 + 0) hwm:85899345920 sw:80530636800, disk sz:37164352012 hwm:53687091200
Later the logs show that the memory usage is now below the stop-writes-pct but the output from asadm does not change.
Jul 26 2016 05:13:17 GMT: INFO (info): (thr_info.c:4872) {problem} memory bytes used 76138494848 (index 76138494848 : sindex 0) : used pct 70.91
====ASCOLLECTINFO====
['namespace']
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace Node Avail% Evictions Master Replica Repl Stop Pending Disk Disk HWM Mem Mem HWM Stop
. . . . Objects Objects Factor Writes Migrates Used Used% Disk% Used Used% Mem% Writes%
. . . . . . . . (tx%,rx%) . . . . . . .
problem server03.customer.com:3000 65 0 14.261M 14.814M 2 true (0,0) 34.612GB 24 50 70.91 GB 70.91 80 75%
Explanation
The reason for this is that, for versions 3.15 and earlier, the stop_writes flag is switched by the namespace supervisor, NSUP. NSUP will check and switch the flag on each run, which would normally happen every 120 seconds (configurable using the nsup-period parameter) but will usually take longer, based on the number of records in the namespace, the number of namespaces and the system performance. If NSUP is taking a long time to run it may be a long time until the flag is switched.
In versions 3.15 and above the stop_writes flag is decoupled from the NSUP thread (was done as part of AER-5160).
The order of tasks for NSUP is as follows:
- Set deletion
- Eviction
- Expiration
For use cases leveraging the set-delete feature (deprecated in versions 3.12 and above), NSUP could be delayed by deleting the sets. NSUP proceeds partition by partition and will not process a new partition until the number of deletes on the delete-queue is less than 10,000. In addition the parameter nsup-delete-sleep which defaults to 100 micro-seconds defines a period for which NSUP will sleep after issuing a delete transaction.
Solution
The key solution is to make sure the cluster is correctly sized to avoid hitting stop-writes. Avoid doing set deletion while in stop-writes as this prolongs the NSUP cycle and means the time spent in stop-writes will be longer. Set high-water-mem-pct to a value to allow the cluster to evict before it hits stop-writes.
Notes
- The flow of set-deletes is discussed here:
+[Now released in version 3.15] - There is an outstanding feature request to decouple stop-writes and NSUP tracked under
- high-water-mem-pct Configuration Reference | Aerospike Documentation
- nsup-period Configuration Reference | Aerospike Documentation
- nsup-delete-sleep Configuration Reference | Aerospike Documentation
- stop-writes-pct Configuration Reference | Aerospike Documentation
Keywords
NSUP DELETE STOP WRITES TRUNCATE
Timestamp
10/02/17