Repl-write ack: from non-dest node

Operations Configuration
1

Hello.

Could you please explain what it means ?

Apr 15 2018 12:05:00 GMT: WARNING (rw): (duplicate_resolve.c:334) dup-res ack: from non-dest node bb9a170d85b8a44
Apr 15 2018 12:05:00 GMT: WARNING (rw): (replica_write.c:373) repl-write ack: from non-dest node bb9a170d85b8a44
Apr 15 2018 12:05:00 GMT: WARNING (rw): (replica_write.c:373) repl-write ack: from non-dest node bb9a170d85b8a44
Apr 15 2018 12:05:00 GMT: WARNING (rw): (duplicate_resolve.c:334) dup-res ack: from non-dest node bb9a170d85b8a44
2

What version of Aerospike are you running?

asadm -e 'info net'
3

I have a cluster on 2 nodes.

Version: 4.0.0.4

4

The server is saying that it is receiving acks from a node that it didn’t expect to have sent a particular type of request to. Since the other node seems to know the in progress transaction-id on the complaining node suggests that the complaining node actually sent the request to the acking node.

You have also observed this issue, Cluster Visibility error, which hints that there may be a cluster configuration issue. Do these nodes have multiple NICs (logical of physical)? If so, could you run ip addr across all node to verify that no two nodes report the same address? You could also try configuring the access-address directly.

5

I have docker on this servers, so I have 172.17.0.1 on both of these servers. But in my network aerospike.conf I specify external ip:

    network {
            service {
                address x.x.x.x
                access-address x.x.x.x

                port 3000
        }
        heartbeat {
                mode mesh
                address x.x.x.x
                port 3002
                interval 150
                timeout 20

                mesh-seed-address-port y.y.y.y 3002
        }
        fabric {
                address x.x.x.x
                port 3001
        }
        info {
                address x.x.x.x
                port 3003
        }
}
6

Ah, docker networking can be tricky. The simplest solution would be to use host networking.

In your configuration, are x.x.x.x or y.y.y.y publicly accessible IP addresses? If they are, you should reconsider running a backed service on a public IP - if not, is really necessary to obfuscate them?

In your config, all x.x.x.x except access-address should be 172.17.0.1 (or any). The y.y.y.y should be the IP of the remote host. Also if the MAC addresses of the 172.17.0.1 interfaces are the same on each container then you will also need to override the node-id.

If you haven’t already, you may want to review our docker guide: https://www.aerospike.com/docs/deploy_guides/docker/networking/index.html

7

It seems I confused you.I have docker thats why I have a lot of ip address on the host node, but i don’t use docker for aerospike.

aerospike is running on the host node. x.x.x.x is real, public (internet accessible) address.

It seems that I fixed the problem. I’m not sure what is the solution, but I edited astools.conf

[cluster]
#host = "localhost:cluster_a:3000"                     # host = [ <host>[:<tls-name>][:<port>], ...]
host = "127.0.0.1:3000"

this is my network config

network {
        service {
                address x.x.x.x
                access-address x.x.x.x
                alternate-access-address x.x.x.x

                port 3000
        }
        heartbeat {
                mode mesh
                address x.x.x.x
                port 3002
                interval 150
                timeout 10

                mesh-seed-address-port y.y.y.y 3002
        }
        fabric {
                address x.x.x.x
                port 3001
        }
        info {
                address x.x.x.x
                port 3003
        }
}

I don’t see any more those messages about: repl-write ack: from non-dest node

Thank you for your help.

8

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.