Adding new node to cluster


#1

Cross-posted here: http://stackoverflow.com/questions/43776536/adding-new-node-to-aerospike-cluster

We are having a two node cluster of aerospike. We thought of adding two more nodes to the cluster. As soon I added them we are getting queue too deep error on new nodes and as well Device over load on client.

I tried of making migrate-max-num-incoming from 256 to 4. Still the issue persists.

What is the best way to add a new node to cluster without impacting the clients.

More info: 1) We are using SSD based installation 2) we are using mesh node architecture


#2

Please provide server version, Community Edition Yes?, namespace configuration, defrag-lwm-pct if set different from default.

Please see this post: Why do I see a warning - "write fail - queue too deep"?


#3

Also, Did you validate your new node’s SSD before adding them in production? Aerospike recommends to do one round of ACT benchmarks for 24 hours before adding any new node in production.


#4

My version is 3.6.3. SSD is Intel S3710. i didn’t set defrag-lwm-pct

I ran act on it and it is satisfactory.

Finally errors went away when I increased max-write-cache to 2047 MB.


#6

You may be able to relive additional disk pressure by increasing your post-write-queue.

Could you also post your namespace configuration?