How will our storage change if we lose 3 nodes in a cluster? (ie how close to high water will we get?)
How much space will we have if we add 2 more nodes?
Both of these questions are pretty straight forward to answer.
If you have 10 nodes and going down to 7 nodes 10/7 = 1.428. Your data usage per node will go up by a factor of 1.428. So if your memory usage is 22% it’ll go up to 22*1.428=31.416 it would then be 31.4%. The same can be applied to memory/dis or bytes/record counts/QPS. Aerospike scales very well so its almost perfect at redistributing the data across nodes. These assume prefer-uniform-balance. I am not sure how this looks in CE, since I just checked and found prefer-uniform-balance is EE feature, but you can always change your node-id to cause data to be rebalanced if you don’t like the way it turned out (node-id shuffling can change partition table succession list!).
The opposite direction works too. If you are starting with 10 nodes and going to 12 your data usage can be multiplied by 10/12 = 0.833 so nodes will go down by that factor. If the servers were at 22% usage each, adding those 2 nodes to a 10-node cluster they would distribute data and each node would end up around (10/12)*22=18.3 18.3% used then instead of 22%.
Just to be perfectly clear this is assuming the total amount of data isn’t changing, and all machines are equal, that only the usage per node is changing. If your app is okay with eviction and you’d reach your HWM it looks different, and if you’re trying to add different sized instances things looks wonky too.