How to add a new node to a cluster approaching utilisation limits (HWM or stop-writes)

Knowledge Base Deploying
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

How to add a new node to a cluster approaching utilisation limits (HWM or stop-writes)

Context

In a situation where a cluster is close to reaching configured upper utilisation limits (high-water-memory-pct, high-water-disk-pct, mounts-high-water-pct, stop-writes-pct or min-avail-pct) it may be necessary to grow the cluster to add capacity to the existing nodes. Adding new nodes in such a case should be done with caution when prefer-uniform-balance is set to true (default in Aerospike server versions from 4.7.x onwards).

Aerospike assigns all data to 4096 master partitions (and a further 4096 partitions for each replica). These partitions are assigned to specific nodes in the cluster automatically. When adding a new node, the partition layout will change. This will result in all nodes migrating their partitions, and therefore data, to each other. This could result in the existing nodes, which are already close to their utilisation limits, to go over said limits if they receive data faster than they manage to migrate it out to the new node(s). In extreme circumstances this could result in nodes hitting stop writes or even being OOM killed.

Method

In order to add new nodes safely in this situation the following action can be taken:

  1. Dynamically set migrate-max-num-incoming to 0 on existing nodes. This will stop any migrations incoming to those nodes.
  2. Add new node(s) to the cluster
  3. Monitor migrations, waiting until the new nodes have received all the partitions they are supposed to hold (migrations incoming - rx - being 0 on the new nodes)
  4. Once all the incoming migrations to the new node(s) have finished, reset the migrate-max-num-incoming value on the existing nodes to the previous value to allow migrations to complete.

Checking current configuration

$ asadm
Admin> show config like migrate-max-num-incoming

~~~~~~~~~~~~~~~~~Service Configuration (2020-05-14 11:35:08 UTC)~~~~~~~~~~~~~~~~~
NODE                    :   09bafa3a326c:3000   172.17.0.4:3000   172.17.0.5:3000   
migrate-max-num-incoming:   4                   4                 4

Stopping migrations on all existing nodes at the same time:

asadm -e 'asinfo -v "set-config:context=service;migrate-max-num-incoming=0"'

Monitoring the remaining migrations:

The below example shows migrates pending (tx,rx) - these are partitions which are still remaining to be migrated out (tx) and in to the node (rx). At this stage, it is necessary to wait until the incoming migrations (rx) on the new nodes show as 0. This will be accompanied by the outgoing migrations (tx) on existing nodes going down as well and their utilisation being reduced.

The following output shows that the node 172.17.0.5 is the new node. Pending Migrates (rx) on that node must reduce to 0.

$ asadm
Admin> watch info namespace


[...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2020-05-14 11:15:30 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace                Node       Total     Repl                            Objects                   Tombstones               Pending   Rack   
        .                   .     Records   Factor         (Master,Prole,Non-Replica)   (Master,Prole,Non-Replica)              Migrates     ID   
        .                   .           .        .                                  .                            .               (tx,rx)      .   
test        09bafa3a326c:3000   100.000 K   2        (49.469 K, 17.428 K, 33.103 K)     (0.000,  0.000,  0.000)      (1.564 K, 0.100 K )      0      
test        172.17.0.4:3000      99.990 K   2        (50.531 K, 16.124 K, 33.335 K)     (0.000,  0.000,  0.000)      (1.466 K, 0.200 K)      0      
test        172.17.0.5:3000      10.000     2        (0.000,  10.000,  0.000)           (0.000,  0.000,  0.000)      (0.000,  2.730 K)     0      
test                            200.000 K            (100.000 K, 33.562 K, 66.438 K)    (0.000,  0.000,  0.000)      (2.930 K, 2.930 K)           
Number of rows: 4
[...]

Set migrate-max-num-incoming on all nodes to previous value

$ asadm -e 'asinfo -v "set-config:context=service;migrate-max-num-incoming=4"'

Keywords

HWM STOP-WRITES GROW CLUSTER ADD NODE MIGRATE-MAX-NUM-INCOMING MAX-NUM-INCOMING LOW DEVICE_AVAIL_PCT MIN-AVAIL-PCT AVAIL-PCT AVAILABLE SCALE

Timestamp

May 2020