Uneven partition distribution

Hi,

I am running Aerospike cluster of 16 nodes in Amazon EC2 with replication factor=1 (no replication). Partition distribution is uneven among Aerospike nodes, with highest=327 and lowest=211.

Cluster reconfiguration (adding/removing nodes) does not resolve this situation. Cluster works fine with this disbalance, but I am afraid it is a symptom of some configuration problem, or one day it’ll become worse and affects overall performance etc.

Should I worry about it? Is there a way to initiate partition re-distribution on a cluster?

Thanks for any help.

Victor

Hi,

Could you, please share, output of asmonitor → info from one of cluster nodes ?

Monitor> info
===NODES===
2015-09-10 13:08:21.071559
Sorting by IP, in Ascending order:
ip:port                                                     Build   Cluster      Cluster   Free   Free   Migrates              Node         Principal   Replicated    Sys
                                                                .      Size   Visibility   Disk    Mem          .                ID                ID      Objects   Free
                                                                .         .            .    pct    pct          .                 .                 .            .    Mem
ec2-52-8-110-117.us-west-1.compute.amazonaws.com:3000      3.5.14        16         true     78     66      (0,0)   BB96B6269306E06   BB9E5EF44048C06    136810559     40
ec2-52-8-112-138.us-west-1.compute.amazonaws.com:3000      3.5.14        16         true     79     67      (0,0)   BB90F4C74730906   BB9E5EF44048C06    132826560     65
ec2-52-8-112-61.us-west-1.compute.amazonaws.com:3000       3.5.14        16         true     79     67      (0,0)   BB9CBA212261E06   BB9E5EF44048C06    128873121     66
ec2-52-8-149-105.us-west-1.compute.amazonaws.com:3000      3.5.14        16         true     78     65      (0,0)   BB933342D9B4A06   BB9E5EF44048C06    138264892     64
ec2-52-8-149-251.us-west-1.compute.amazonaws.com:3000      3.5.14        16         true     81     71      (0,0)   BB98F265C8B9A06   BB9E5EF44048C06    115453187     70
ec2-52-8-19-167.us-west-1.compute.amazonaws.com:3000       3.5.14        16         true     80     68      (0,0)   BB9A9D996918E06   BB9E5EF44048C06    127351676     67
ec2-52-8-32-134.us-west-1.compute.amazonaws.com:3000       3.5.14        16         true     77     64      (0,0)   BB9E5EF44048C06   BB9E5EF44048C06    143704083     63
ec2-52-8-40-142.us-west-1.compute.amazonaws.com:3000       3.5.14        16         true     80     68      (0,0)   BB92D66F387A906   BB9E5EF44048C06    127344484     67
ec2-52-8-54-252.us-west-1.compute.amazonaws.com:3000       3.5.14        16         true     79     67      (0,0)   BB9CF548FD02406   BB9E5EF44048C06    129826120     66
ec2-52-8-7-111.us-west-1.compute.amazonaws.com:3000        3.5.14        16         true     79     67      (0,0)   BB9B1A66FAC8606   BB9E5EF44048C06    129355465     66
ec2-52-8-78-116.us-west-1.compute.amazonaws.com:3000       3.5.14        16         true     81     70      (0,0)   BB931067CE7B106   BB9E5EF44048C06    120413337     68
ec2-52-8-81-13.us-west-1.compute.amazonaws.com:3000        3.5.14        16         true     81     70      (0,0)   BB9D99561826E06   BB9E5EF44048C06    118459652     69
ec2-52-8-94-167.us-west-1.compute.amazonaws.com:3000       3.5.14        16         true     82     72      (0,0)   BB951ED18694706   BB9E5EF44048C06    112024402     70
ec2-54-153-56-227.us-west-1.compute.amazonaws.com:3000     3.5.14        16         true     82     71      (0,0)   BB9756165753206   BB9E5EF44048C06    114000625     70
ec2-54-183-117-136.us-west-1.compute.amazonaws.com:3000    3.5.14        16         true     80     69      (0,0)   BB9DDB05D697306   BB9E5EF44048C06    122920090     68
ec2-54-183-150-10.us-west-1.compute.amazonaws.com:3000     3.5.14        16         true     79     67      (0,0)   BB9DFB968B4CE06   BB9E5EF44048C06    132316872     66
Number of nodes displayed: 16


 ===NAMESPACE===
Total (unique) objects in cluster for namespace : 2029945119
Note: Total (unique) objects is an under estimate if migrations are in progress.


ip/namespace                                                    Avail    Evicted      Master     Repl     Stop      Used   Used     Used   Used    hwm   hwm
                                                                  Pct    Objects     Objects   Factor   Writes      Disk   Disk      Mem    Mem   Disk   Mem
                                                                    .          .           .        .        .         .      %        .      %      .     .
ec2-52-8-32-134.us-west-1.compute.amazonaws.com/namespace          61          0   143704083        1    false   68.71 G     23   8.57 G     36     50    60
ec2-52-8-110-117.us-west-1.compute.amazonaws.com/namespace         63   58707432   136810559        1    false   65.32 G     22   8.15 G     34     50    60
ec2-52-8-149-105.us-west-1.compute.amazonaws.com/namespace         63          0   138264897        1    false   66.09 G     22   8.24 G     35     50    60
ec2-54-183-150-10.us-west-1.compute.amazonaws.com/namespace        64          0   132316872        1    false   63.25 G     21   7.89 G     33     50    60
ec2-52-8-112-138.us-west-1.compute.amazonaws.com/namespace         64          0   132826550        1    false   63.49 G     21   7.92 G     33     50    60
ec2-52-8-54-252.us-west-1.compute.amazonaws.com/namespace          65          0   129826122        1    false   62.05 G     21   7.74 G     33     50    60
ec2-52-8-7-111.us-west-1.compute.amazonaws.com/namespace           65          0   129355468        1    false   61.84 G     21   7.71 G     33     50    60
ec2-52-8-112-61.us-west-1.compute.amazonaws.com/namespace          65          0   128873111        1    false   61.61 G     21   7.68 G     33     50    60
ec2-52-8-40-142.us-west-1.compute.amazonaws.com/namespace          66          0   127344490        1    false   60.89 G     20   7.59 G     32     50    60
ec2-52-8-19-167.us-west-1.compute.amazonaws.com/namespace          66          0   127351682        1    false   60.88 G     20   7.59 G     32     50    60
ec2-54-183-117-136.us-west-1.compute.amazonaws.com/namespace       67          0   122920079        1    false   58.76 G     20   7.33 G     31     50    60
ec2-52-8-81-13.us-west-1.compute.amazonaws.com/namespace           68          0   118459655        1    false   56.63 G     19   7.06 G     30     50    60
ec2-52-8-78-116.us-west-1.compute.amazonaws.com/namespace          68          0   120413336        1    false   57.56 G     19   7.18 G     30     50    60
ec2-52-8-149-251.us-west-1.compute.amazonaws.com/namespace         69          0   115453193        1    false   55.20 G     19   6.88 G     29     50    60
ec2-54-153-56-227.us-west-1.compute.amazonaws.com/namespace        69          0   114000620        1    false   54.49 G     18   6.79 G     29     50    60
ec2-52-8-94-167.us-west-1.compute.amazonaws.com/namespace          70          0   112024402        1    false   53.55 G     18   6.68 G     28     50    60
Number of rows displayed: 16
Monitor>

Check the prefer-uniform-balance configuration which should distribute the partitions evenly. The default may yield uneven distributions but optimizes for partitions movement during migrations.