How to add a new node to a cluster approaching utilisation limits (HWM or stop-writes)
In a situation where a cluster is close to reaching configured upper utilisation limits (
stop-writes-pct or min-avail-pct) it may be necessary to grow the cluster to add capacity to the existing nodes. Adding new nodes in such a case should be done with caution when
prefer-uniform-balance is set to
true (default in Aerospike server versions from 4.7.x onwards).
Aerospike assigns all data to 4096 master partitions (and a further 4096 partitions for each replica). These partitions are assigned to specific nodes in the cluster automatically. When adding a new node, the partition layout will change. This will result in all nodes migrating their partitions, and therefore data, to each other. This could result in the existing nodes, which are already close to their utilisation limits, to go over said limits if they receive data faster than they manage to migrate it out to the new node(s). In extreme circumstances this could result in nodes hitting stop writes or even being OOM killed.
In order to add new nodes safely in this situation the following action can be taken:
- Dynamically set
migrate-max-num-incomingto 0 on existing nodes. This will stop any migrations incoming to those nodes.
- Add new node(s) to the cluster
- Monitor migrations, waiting until the new nodes have received all the partitions they are supposed to hold (migrations incoming - rx - being 0 on the new nodes)
- Once all the incoming migrations to the new node(s) have finished, reset the
migrate-max-num-incomingvalue on the existing nodes to the previous value to allow migrations to complete.
Checking current configuration
$ asadm Admin> show config like migrate-max-num-incoming ~~~~~~~~~~~~~~~~~Service Configuration (2020-05-14 11:35:08 UTC)~~~~~~~~~~~~~~~~~ NODE : 09bafa3a326c:3000 172.17.0.4:3000 172.17.0.5:3000 migrate-max-num-incoming: 4 4 4
Stopping migrations on all existing nodes at the same time:
asadm -e 'asinfo -v "set-config:context=service;migrate-max-num-incoming=0"'
Monitoring the remaining migrations:
The below example shows migrates pending (tx,rx) - these are partitions which are still remaining to be migrated out (tx) and in to the node (rx). At this stage, it is necessary to wait until the incoming migrations (rx) on the new nodes show as 0. This will be accompanied by the outgoing migrations (tx) on existing nodes going down as well and their utilisation being reduced.
The following output shows that the node
172.17.0.5 is the new node. Pending Migrates (rx) on that node must reduce to 0.
$ asadm Admin> watch info namespace [...] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2020-05-14 11:15:30 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Namespace Node Total Repl Objects Tombstones Pending Rack . . Records Factor (Master,Prole,Non-Replica) (Master,Prole,Non-Replica) Migrates ID . . . . . . (tx,rx) . test 09bafa3a326c:3000 100.000 K 2 (49.469 K, 17.428 K, 33.103 K) (0.000, 0.000, 0.000) (1.564 K, 0.100 K ) 0 test 172.17.0.4:3000 99.990 K 2 (50.531 K, 16.124 K, 33.335 K) (0.000, 0.000, 0.000) (1.466 K, 0.200 K) 0 test 172.17.0.5:3000 10.000 2 (0.000, 10.000, 0.000) (0.000, 0.000, 0.000) (0.000, 2.730 K) 0 test 200.000 K (100.000 K, 33.562 K, 66.438 K) (0.000, 0.000, 0.000) (2.930 K, 2.930 K) Number of rows: 4 [...]
Set migrate-max-num-incoming on all nodes to previous value
$ asadm -e 'asinfo -v "set-config:context=service;migrate-max-num-incoming=4"'
HWM STOP-WRITES GROW CLUSTER ADD NODE MIGRATE-MAX-NUM-INCOMING