We faced with the problem: scan/query all records returns only half of the records after an any node restarting
How to reproduce:
Deploy aerospike cluster in google cloud with 2 nodes as described here: https://cloud.google.com/launcher/solution/click-to-deploy-images/aerospike
Create a simple application with latest Aerospike java client (in my case “3.1.9”) which connects to the cluster and creates 10 records
Every second application queries all records from the set (or scans the set) and expects to receive 10 records
Manually restart one the aerospike nodes. After the restart we will see two possible results:
a) We receive 15 records instead of 10. But in a second this problem disappear
b) We receive 5 records instead if 10. Problem repeats every second. It disappers only when we restart application or recreate AerospikeClient object.
I see why the case “a” happens after the restart (it was described in one of the posts on this forum). But why the problem does not disappear in case “b” until we restart an app or recreate AerospikeClient object again?
UPD: it happens only if app server with java app is located far enough from aerospike cluster (in another region). It is not reproduced if app server is located in the same datacenter with Aerospike.