Why Aerospike evicted data?

Operations Configuration
1

Hi all! I have some problem with entering data into aerospike. I have 2 node in cluster, with similar performance. Config file nodes: (i think problem in section namespace)

namespace asg {
        replication-factor 2
        high-water-memory-pct 70
        high-water-disk-pct 90
        stop-writes-pct 70
        memory-size 4294967296
        default-ttl 864000
 
        storage-engine device {
                file /opt/citrusleaf/data/test.data
                filesize 107374182400 # 100G
               data-in-memory true
               defrag-period 120
              defrag-lwm-pct 50
              defrag-max-blocks 4000
             defrag-startup-minimum 10
        }
}

When i entering data, after the amount of data comes to 16,15 Gb, Aerospike start evicted data.Why? This is a report of clmonitor

 ===NAMESPACE===
Total(unique) objects in cluster for asg : 33.835 M
ip/namespace              Avail   Evicted    Objects     Repl     Stop      Used   Used     Used   Used    hwm   hwm
                            Pct   Objects          .   Factor   Writes      Disk   Disk      Mem    Mem   Disk   Mem
                              .         .          .        .        .         .      %        .      %      .     .
192.168.180.192/asg        72   269,110   16.923 M        2     true   16.14 G     17   2.81 G     71     90    70
192.168.180.193/asg       72   169,997   16.912 M        2     true   16.13 G     17   2.80 G     71     90    70
Number of rows displayed: 2
Every 1.0s: info                        2013-06-11 15:34:00.997859
===NODES===
2013-06-11 15:34:00.998202
ip:port               Build   Cluster      Cluster   Free   Free   Migrates              Node   Replicated    Sys
                          .      Size   Visibility   Disk    Mem          .                id      Objects   Free
                          .         .            .    pct    pct          .                 .            .    Mem
192.168.180.193    2.6.7         2         true     83     29      (0,0)   BB99066C13539E8     33.817 M     89
192.168.180.192     2.6.7         2         true     83     29      (0,0)   BB93090686EC180     33.837 M     85
Number of nodes displayed: 2
 
 
===NAMESPACE===
Total(unique) objects in cluster for asg : 33.827 M
ip/namespace              Avail   Evicted    Objects     Repl     Stop      Used   Used     Used   Used    hwm   hwm
                            Pct   Objects          .   Factor   Writes      Disk   Disk      Mem    Mem   Disk   Mem
                              .         .          .        .        .         .      %        .      %      .     .
192.168.180.192/asg        72   269,110   16.919 M        2     true   16.13 G     17   2.80 G     71     90    70
192.168.180.193/asg       72   169,997   16.909 M        2     true   16.13 G     17   2.80 G     71     90    70
Number of rows displayed: 2

I understand that the problem is due to the setting namespases, but do not know what exactly. Please help me understand p.s. Sorry for my English, i used translate.google.com :lol: Best regards!

2

Thank you for your submission. Let me start by explaining what “eviction” means and I hope that Google translate will do a good job.

First, let me explain “expiration.” Many databases have this today. This is set by the TTL (time to live). Any object in the database that is past its TTL will be automatically deleted from the database. For example if the TTL is set at 90 days, any data longer than 90 days will be deleted.

Now comes “eviction”. As a safety feature, Aerospike has settings that will “evict” or get rid of data once the database is getting close to full. Keep in mind that as a distributed database, you want to have extra capacity so that if a node goes down, the remaining nodes much be able to handle all the traffic. By default, we recommend that you run so that the disk not be more than 50% full (this is also for defragmentation) and memory is no more than 60% full. These are controlled by the “high-water-memory-pct” and “high-water-disk-pct”.

In addition, there is a level at which the database is considered so full that it will stop writing any new records (updates to existing records are still possible). This is called the stop-write level.

These settings are determined by the following variables in your configuration files:

high-water-memory-pct 70 high-water-disk-pct 90 stop-writes-pct 70

Your current high-water-memory-pct is 70% and according to the clmonitor output, you are at 71%. So this means that you are above the memory high water mark. In addition, you are above the stop-writes-pct, so the database is not accepting new records (the “Stop Writes” is “true” in the clmonitor output).

There are 3 ways around this:

  1. Reduce the number of records/amount of data
  2. Increase the memory allocated for the database (“memory-size 4294967296”)
  3. Increase the limits. Increasing the limits should only be seen as temporary measure.

Please refer to the documentation at: https://docs.aerospike.com/display/AS2/Managing+Storage+Capacity

2 Likes
3

The link is expired. Check this link for the the documentation