We’re in the middle of transitioning to some new hardware, and as part of the process I have to remove a few old nodes from our cluster. I’ve been careful to remove nodes at the right time (e.g., removing one node at a time, waiting for migrates to finish, etc.), but whenever I remove a node it seems to cause all of the Aerospike clients that are reading from the cluster to temporarily throw exceptions like this:
com.aerospike.client.AerospikeException$Timeout: Client timeout: timeout=0 iterations=2 failedNodes=0 failedConns=2 at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:131) at com.aerospike.client.command.BatchExecutor.execute(BatchExecutor.java:53) at com.aerospike.client.AerospikeClient.get(AerospikeClient.java:606)
What is the preferred way to remove a node from the cluster in such a way that it is transparent to currently active clients?