I am testing a new cluster on AWS, single AZ, with placement group. I started with a cluster of size 20 and decided to decrease it to size 2 (I didn’t care for lost data since this is testing). I shut down all but two instances, however, they never updated the cluster size correctly:
ip:port Build Cluster Cluster Free Free Migrates Node Principal Replicated Sys . Size Visibility Disk Mem . ID ID Objects Free . . . pct pct . . . . Mem ip-172-30-29-77.ec2.internal:3000 3.6.4 10 false 96 79 (219,3) BB98D36254C080A BB9B55D74EA770A 181,503,188 80 ip-172-30-30-245.ec2.internal:3000 3.6.4 10 false 96 80 (210,0) BB9B55D74EA770A BB9B55D74EA770A 179,681,969 80 Number of nodes displayed: 2
I also tried running service-alumni-reset but that didn’t seem to help. Any idea what’s wrong?
The only way I was able to getting back in sync was by bouncing both nodes, which with cold restart is not really a great option…