Go client brings partial data when got disconnected from a node


#1

Hi, I’m running an AS cluster with 2 nodes and replication factor of 2 on GCE. The cluster is configured to use mesh over private network for internode communication. When I connect to the cluster from outside, supplying one seed using it’s public IP and then run a scan, it brings only master copies stored on that node. When I shut the other node down and run the scan again, it brings the whole data (master + replica) as expected.

Please advise if it’s a bug or there is some workaround.

Thanks, Dmitri.


#2

If that’s the case, it’s should be a bug. Let me investigate and get back to you.

Could you provide your cluster configuration? (or is it possible to use your cluster for debugging?)


#3

Thank you for getting back. Actually my configuration is pretty standard (please see below):

# AS configuration file

service {
        user root
        group root
        paxos-single-replica-limit 1
        pidfile /var/run/aerospike/asd.pid
        service-threads 4
        transaction-queues 4
        transaction-threads-per-queue 4
        proto-fd-max 15000
}

logging {
        # Log file must be an absolute path.
        file /var/log/aerospike/aerospike.log {
                context any info
        }
}

network {
        service {
                address any
                port 3000
        }
        heartbeat {
                mode mesh
                address ${MY_INTERNAL_IP}  # IP of the NIC on which this node is listening
                port 3002

                # IP address for seed node in the cluster
                mesh-seed-address-port ${MY_INTERNAL_IP} 3002
                mesh-seed-address-port ${OTHER_NODE_INTERNAL_IP} 3002

                interval 150
                timeout 20
        }
        fabric {
                port 3001
        }
        info {
                port 3003
        }
}

namespace ${MY_NS} {
        replication-factor 2
        memory-size YG
        default-ttl 0
        ldt-enabled true
        storage-engine device {
               file /opt/aerospike/data/${MY_NS}.dat
               filesize XG
               disable-odirect true
               data-in-memory true
       }

Unfortunately, it’s impossible to open the cluster for debugging. But I may provide you with steps how to reproduce the issue. As for the versions, I use Debian 7.9 and AS 3.7.4 community edition.

Thank you, Dmitri.