Namespace stays in stop-writes even though memory is less than stop-writes-pct


#1

Descriptive title of issue

Aerospike Server version 3.12.1 or above

As of version 3.12.1, released in April 2017, set-delete is deprecated. Truncate the new feature of deleting all the data in a namespace or set is now supported in the database.

Using the truncate command will not prolong stop-writes.

See the following truncate documentation: http://www.aerospike.com/docs/reference/info#truncate

Truncate can also be executed from the client APIs. Here is the java documentation: http://www.aerospike.com/apidocs/java/com/aerospike/client/AerospikeClient.html

Aerospike Server version 3.12.1 or below

Problem Description

A namespace goes into stop-writes as the memory usage exceeds the stop-writes-pct threshold. Even though subsequently the memory drops below the threshold, the stop-writes remains set when the namespace is viewed in asadm

====ASCOLLECTINFO====
['namespace']
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Namespace                               Node   Avail%   Evictions      Master     Replica     Repl     Stop     Pending         Disk    Disk     HWM          Mem     Mem    HWM      Stop   
          .                                  .        .           .     Objects     Objects   Factor   Writes    Migrates         Used   Used%   Disk%         Used   Used%   Mem%   Writes%   
          .                                  .        .           .           .           .        .        .   (tx%,rx%)            .       .       .            .       .      .         .   
    
problem              server03.customer.com:3000  65             0      14.261M    14.814M      2       true      (0,0)       34.612GB   24       50         75.01 GB  75.01      80      75%

Initially the logs show that stop-writes is true.

Jul 26 2016 01:32:01 GMT: WARNING (namespace): (namespace.c:440) {problem} hwm_breached false, stop_writes true (memory), memory sz:80548665920 (80548665920 + 0) hwm:85899345920 sw:80530636800, disk sz:37164352012 hwm:53687091200

Later the logs show that the memory usage is now below the stop-writes-pct but the output from asadm does not change.

Jul 26 2016 05:13:17 GMT: INFO (info): (thr_info.c:4872) {problem} memory bytes used 76138494848 (index 76138494848 : sindex 0) : used pct 70.91
====ASCOLLECTINFO====
['namespace']
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Namespace                               Node   Avail%   Evictions      Master     Replica     Repl     Stop     Pending         Disk    Disk     HWM          Mem     Mem    HWM      Stop   
          .                                  .        .           .     Objects     Objects   Factor   Writes    Migrates         Used   Used%   Disk%         Used   Used%   Mem%   Writes%   
          .                                  .        .           .           .           .        .        .   (tx%,rx%)            .       .       .            .       .      .         .   
    
problem              server03.customer.com:3000  65             0      14.261M    14.814M      2       true      (0,0)       34.612GB   24       50         70.91 GB  70.91      80      75%

Explanation

The reason for this is that, for versions 3.15 and earlier, the stop_writes flag is switched by the namespace supervisor, NSUP. NSUP will check and switch the flag on each run, which would normally happen every 120 seconds (configurable using the nsup-period parameter) but will usually take longer, based on the number of records in the namespace, the number of namespaces and the system performance. If NSUP is taking a long time to run it may be a long time until the flag is switched.

In versions 3.15 and above the stop_writes flag is decoupled from the NSUP thread (was done as part of AER-5160).

The order of tasks for NSUP is as follows:

  1. Set deletion
  2. Eviction
  3. Expiration

For use cases leveraging the set-delete feature (deprecated in versions 3.12 and above), NSUP could be delayed by deleting the sets. NSUP proceeds partition by partition and will not process a new partition until the number of deletes on the delete-queue is less than 10,000. In addition the parameter nsup-delete-sleep which defaults to 100 micro-seconds defines a period for which NSUP will sleep after issuing a delete transaction.

Solution

The key solution is to make sure the cluster is correctly sized to avoid hitting stop-writes. Avoid doing set deletion while in stop-writes as this prolongs the NSUP cycle and means the time spent in stop-writes will be longer. Set high-water-mem-pct to a value to allow the cluster to evict before it hits stop-writes.

Notes

  • The flow of set-deletes is discussed here:

+[Now released in version 3.15] - There is an outstanding feature request to decouple stop-writes and NSUP tracked under

Keywords

NSUP DELETE STOP WRITES TRUNCATE

Timestamp

10/02/17


Understanding when server no longer accepts writes