Aerospike Go client: timed out issue

Operations Tuning
1

I am using Aerospike go-client (v1.24.0). I have 10 node aerospike cluster (v3.8.1). I am getting frequent error message -

E0621 09:59:01.457130 2514 Aerospike.go:32] Unable to fetch from Aerospike : command execution timed out: Exceeded number of retries. See Policy.MaxRetries. (last error: No available connections to the node. Connection Pool was empty, and limited to certain number of connections.)

There are some read connection timeout logs also:

E0621 10:22:06.522898 22038 Aerospike.go:32] Unable to fetch from Aerospike : read tcp 10.125.4.254:46096->10.57.115.42:3000: read: connection timed out

I have noticed that this error always has 10.57.115.42 IP. When i checked on AMC, the number of client connection to this node/IP(10.57.115.42) is double (~8k) than other nodes (~4k).

I am using default client policy in aerospike-go-client.

Node config:

service {
        user root
        group root
        paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
        pidfile /var/run/aerospike/asd.pid
        service-threads 24
        transaction-queues 24
        transaction-threads-per-queue 4
        proto-fd-max 30000
        transaction-pending-limit 0
}

What could be the issue?

2

Any update on this Guys? we are frequently facing this issues .

3

Hi Nishant, command execution timed out: Exceeded number of retries. SeePolicy.MaxRetries = this error comes when the timeout happens based on the readpolicy you have specified during GET operation . do check about “aerospike.BasePolicy” in case you have used default configuration

And no idea about read: connection timeout :slight_smile: