How to troubleshoot warnings in Aerospike logs after node restart

Operations Configuration
1

Hi, I have a 2 node Aerospike cluster hosted on EC2 instances. The Aerospike is running as an independent process on the EC2 instances but they also have unrelated Docker containers running on them. I changed some configuration and executed a rolling restart of the 2 nodes. I see below lines in aerospike.log:

Nov 24 2020 18:52:25 GMT: WARNING (rw): (replica_write.c:331) repl-write ack: from non-dest node bb91fa1d31d8902
Nov 24 2020 18:52:26 GMT: WARNING (rw): (duplicate_resolve.c:290) dup-res ack: from non-dest node bb91fa1d31d8902
Nov 24 2020 18:52:26 GMT: WARNING (migrate): (migrate.c:1582) ctrl ack (14): unexpected source bb91fa1d31d8902
Nov 24 2020 18:52:27 GMT: WARNING (rw): (duplicate_resolve.c:290) dup-res ack: from non-dest node bb91fa1d31d8902
Nov 24 2020 18:52:27 GMT: WARNING (rw): (duplicate_resolve.c:290) dup-res ack: from non-dest node bb91fa1d31d8902
Nov 24 2020 18:52:28 GMT: WARNING (rw): (replica_write.c:331) repl-write ack: from non-dest node bb91fa1d31d8902
Nov 24 2020 18:52:28 GMT: WARNING (rw): (duplicate_resolve.c:290) dup-res ack: from non-dest node bb91fa1d31d8902
Nov 24 2020 18:52:28 GMT: WARNING (partition): (partition_balance.c:1512) {tala:2762} immigrate_start - source bb91fa1d31d8902 working-master bb91fa1d31d8902 pending-immigrations 0 - no immigrations expected

asinfo -v services shows the EC2’s private IP address plus 172.17.0.1

asinfo -v services
10.10.2.146:3000;172.17.0.1:3000

Here is my aerospike.conf

network {
service {
address any
port 3000
}

heartbeat {
address any
mode mesh
port 3002 # Heartbeat port for this node.

  # List one or more other nodes, one ip-address & port per line:

  mesh-seed-address-port 10.10.1.150 3002
  mesh-seed-address-port 10.10.2.146 3002
  interval 150
  timeout 10

}

How do I fix the warnings in the logs? The cluster shows up as having only 2 nodes but with each node having the 2 addresses…

2

You may simply be missing the access-address directive to set the IP (interface) to be used by the client application. Otherwise, by default, nodes would publish all interface IPs they can see.

Those WARNINGs indicate the server is receiving acks from a node that it didn’t expect to have sent a particular type of request to. Since the other node seems to know the in progress transaction-id on the complaining node suggests that the complaining node actually sent the request to the acking node. I think this would go along with the theory of duplicate IP addresses per node or somehow nodes from the containers getting in the way?

Maybe the following article can help too? https://www.aerospike.com/docs/deploy_guides/docker/networking/index.html