Hi,
I am running Aerospike cluster of 16 nodes in Amazon EC2 with replication factor=1 (no replication). Partition distribution is uneven among Aerospike nodes, with highest=327 and lowest=211.
Cluster reconfiguration (adding/removing nodes) does not resolve this situation. Cluster works fine with this disbalance, but I am afraid it is a symptom of some configuration problem, or one day it’ll become worse and affects overall performance etc.
Should I worry about it? Is there a way to initiate partition re-distribution on a cluster?
Thanks for any help.
Victor
Vishnu
August 25, 2015, 1:48pm
2
Hi,
Could you, please share, output of asmonitor → info from one of cluster nodes ?
Monitor> info
===NODES===
2015-09-10 13:08:21.071559
Sorting by IP, in Ascending order:
ip:port Build Cluster Cluster Free Free Migrates Node Principal Replicated Sys
. Size Visibility Disk Mem . ID ID Objects Free
. . . pct pct . . . . Mem
ec2-52-8-110-117.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 78 66 (0,0) BB96B6269306E06 BB9E5EF44048C06 136810559 40
ec2-52-8-112-138.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 79 67 (0,0) BB90F4C74730906 BB9E5EF44048C06 132826560 65
ec2-52-8-112-61.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 79 67 (0,0) BB9CBA212261E06 BB9E5EF44048C06 128873121 66
ec2-52-8-149-105.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 78 65 (0,0) BB933342D9B4A06 BB9E5EF44048C06 138264892 64
ec2-52-8-149-251.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 81 71 (0,0) BB98F265C8B9A06 BB9E5EF44048C06 115453187 70
ec2-52-8-19-167.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 80 68 (0,0) BB9A9D996918E06 BB9E5EF44048C06 127351676 67
ec2-52-8-32-134.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 77 64 (0,0) BB9E5EF44048C06 BB9E5EF44048C06 143704083 63
ec2-52-8-40-142.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 80 68 (0,0) BB92D66F387A906 BB9E5EF44048C06 127344484 67
ec2-52-8-54-252.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 79 67 (0,0) BB9CF548FD02406 BB9E5EF44048C06 129826120 66
ec2-52-8-7-111.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 79 67 (0,0) BB9B1A66FAC8606 BB9E5EF44048C06 129355465 66
ec2-52-8-78-116.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 81 70 (0,0) BB931067CE7B106 BB9E5EF44048C06 120413337 68
ec2-52-8-81-13.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 81 70 (0,0) BB9D99561826E06 BB9E5EF44048C06 118459652 69
ec2-52-8-94-167.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 82 72 (0,0) BB951ED18694706 BB9E5EF44048C06 112024402 70
ec2-54-153-56-227.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 82 71 (0,0) BB9756165753206 BB9E5EF44048C06 114000625 70
ec2-54-183-117-136.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 80 69 (0,0) BB9DDB05D697306 BB9E5EF44048C06 122920090 68
ec2-54-183-150-10.us-west-1.compute.amazonaws.com:3000 3.5.14 16 true 79 67 (0,0) BB9DFB968B4CE06 BB9E5EF44048C06 132316872 66
Number of nodes displayed: 16
===NAMESPACE===
Total (unique) objects in cluster for namespace : 2029945119
Note: Total (unique) objects is an under estimate if migrations are in progress.
ip/namespace Avail Evicted Master Repl Stop Used Used Used Used hwm hwm
Pct Objects Objects Factor Writes Disk Disk Mem Mem Disk Mem
. . . . . . % . % . .
ec2-52-8-32-134.us-west-1.compute.amazonaws.com/namespace 61 0 143704083 1 false 68.71 G 23 8.57 G 36 50 60
ec2-52-8-110-117.us-west-1.compute.amazonaws.com/namespace 63 58707432 136810559 1 false 65.32 G 22 8.15 G 34 50 60
ec2-52-8-149-105.us-west-1.compute.amazonaws.com/namespace 63 0 138264897 1 false 66.09 G 22 8.24 G 35 50 60
ec2-54-183-150-10.us-west-1.compute.amazonaws.com/namespace 64 0 132316872 1 false 63.25 G 21 7.89 G 33 50 60
ec2-52-8-112-138.us-west-1.compute.amazonaws.com/namespace 64 0 132826550 1 false 63.49 G 21 7.92 G 33 50 60
ec2-52-8-54-252.us-west-1.compute.amazonaws.com/namespace 65 0 129826122 1 false 62.05 G 21 7.74 G 33 50 60
ec2-52-8-7-111.us-west-1.compute.amazonaws.com/namespace 65 0 129355468 1 false 61.84 G 21 7.71 G 33 50 60
ec2-52-8-112-61.us-west-1.compute.amazonaws.com/namespace 65 0 128873111 1 false 61.61 G 21 7.68 G 33 50 60
ec2-52-8-40-142.us-west-1.compute.amazonaws.com/namespace 66 0 127344490 1 false 60.89 G 20 7.59 G 32 50 60
ec2-52-8-19-167.us-west-1.compute.amazonaws.com/namespace 66 0 127351682 1 false 60.88 G 20 7.59 G 32 50 60
ec2-54-183-117-136.us-west-1.compute.amazonaws.com/namespace 67 0 122920079 1 false 58.76 G 20 7.33 G 31 50 60
ec2-52-8-81-13.us-west-1.compute.amazonaws.com/namespace 68 0 118459655 1 false 56.63 G 19 7.06 G 30 50 60
ec2-52-8-78-116.us-west-1.compute.amazonaws.com/namespace 68 0 120413336 1 false 57.56 G 19 7.18 G 30 50 60
ec2-52-8-149-251.us-west-1.compute.amazonaws.com/namespace 69 0 115453193 1 false 55.20 G 19 6.88 G 29 50 60
ec2-54-153-56-227.us-west-1.compute.amazonaws.com/namespace 69 0 114000620 1 false 54.49 G 18 6.79 G 29 50 60
ec2-52-8-94-167.us-west-1.compute.amazonaws.com/namespace 70 0 112024402 1 false 53.55 G 18 6.68 G 28 50 60
Number of rows displayed: 16
Monitor>
meher
June 13, 2019, 1:35pm
4
Check the prefer-uniform-balance configuration which should distribute the partitions evenly. The default may yield uneven distributions but optimizes for partitions movement during migrations.