- 4 nodes aerospike cluster with replication factor of 2
- a set with 670 million records within the cluster
- 7 bins (4 strings, 3 integers) in every record of the set
- and a string s-index on one of the string bins (the bin’s data is always a 36 symbols string of one of 350 million unique values distributed uniformaly between those 670 millions).
After dropping the s-index the entire cluster stopped responding, clients couldn’t connect to the aerospike nodes, asmonitor kept printing timeout errors.
It lasted for about 10 minutes. During that period we could see the s-index usage of ram was slowly going down until reached 0.
After that the cluster went back to normal, the state of the cluster remained unchanged, migrations didn’t start.
Why is this happening?