Cluster size not updating

1

I am testing a new cluster on AWS, single AZ, with placement group. I started with a cluster of size 20 and decided to decrease it to size 2 (I didn’t care for lost data since this is testing). I shut down all but two instances, however, they never updated the cluster size correctly:

ip:port                               Build   Cluster      Cluster   Free   Free   Migrates              Node         Principal    Replicated    Sys
                                          .      Size   Visibility   Disk    Mem          .                ID                ID       Objects   Free
                                          .         .            .    pct    pct          .                 .                 .             .    Mem
ip-172-30-29-77.ec2.internal:3000     3.6.4        10        false     96     79    (219,3)   BB98D36254C080A   BB9B55D74EA770A   181,503,188     80
ip-172-30-30-245.ec2.internal:3000    3.6.4        10        false     96     80    (210,0)   BB9B55D74EA770A   BB9B55D74EA770A   179,681,969     80
Number of nodes displayed: 2

I also tried running service-alumni-reset but that didn’t seem to help. Any idea what’s wrong?

The only way I was able to getting back in sync was by bouncing both nodes, which with cold restart is not really a great option…

2

This should definitely not be the expected behavior. We will test this on our side.

Did you explicitly stop Aerospike service on the nodes before shutting down the instances.

sudo service aerospike stop

If so, did you see the cluster size dropping as you stopped the service?