Client.get() timeout after a couple of requests


#1

Hi,

I am evaluating aerospike for a realtime, hi throughput application. Basically, I have a 2-node cluster with a pair of java applications writing data into aerospike as they come in from a high volume data stream, and then a third application will expose these data via a RESTful interface.

I made a simple application to keep querying the same key over and over again, to test performances, but I can get just the first two replies, then the third loop will get stuck (indefinitely):

public class QueryKey {

    public static void main(String[] args) throws PropertyException {

        // Some initialization here...

        QueryKey qk = new QueryKey(args);
        qk.query();
        qk.close();

    }

    final String ns;
    final String set;
    final String key;
    final AerospikeClient ac;


    public QueryKey(final String[] args) throws PropertyException {
        this.ns = args[0];
        this.set = args[1];
        this.key = args[2];

        ClientPolicy policy = new ClientPolicy();
        policy.timeout = 3000;
        policy.maxConnsPerNode = 1;
        policy.readPolicyDefault.timeout = 3000;
        this.ac = new AerospikeClient(
                policy,
                new Host[] {
                        new Host("10.0.0.11", 3000),
                        new Host("10.0.0.12", 3000)
                        }
                );
    }


    public void query() {
        long reqNo = 1;
        long before = 0;
        while (true) {
            before = System.currentTimeMillis();
            System.out.println("Req: " + reqNo++ + " -- " + before);
            Record r = ac.get(null, new Key(ns, set, key));
                for (String k : r.bins.keySet()) {
                    System.out.println(k + " => " + new String( (byte[]) r.bins.get(k), Charset.forName("UTF-8")));
                }
            System.out.println("Req exc time: " + (System.currentTimeMillis() - before) + "ms");
        }
    }


    public void close() {
        ac.close();
    }

}

The output stops on the third request:

Req: 1 -- 1467986439188
    [ various key => value pairs...]
Req exc time: 35ms
Req: 2 -- 1467986439223
    [ various key => value pairs...]
Req exc time: 19ms
Req: 3 -- 1467986439242
    Exception in thread "main" com.aerospike.client.AerospikeException$Timeout: Client timeout: timeout=3000 iterations=1 failedNodes=0 failedConns=0 lastNode=BB9000100000001 10.0.0.11:3000
        at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:86)
        at com.aerospike.client.AerospikeClient.get(AerospikeClient.java:508)
        at it.webank.info.cache.feeder.test.QueryKey.query(QueryKey.java:71)
        at it.webank.info.cache.feeder.test.QueryKey.main(QueryKey.java:26)

EDIT: if I change slightly the loop it clearly shows that 2 requests go well, and 2 fail, alternatively:

        public void query() {
            long reqNo = 1;
            long before = 0;
            Record r = null;
            while (true) {
                try {
                    before = System.currentTimeMillis();
                    log("Req: " + reqNo++ + " -- " + before);
                    r = ac.get(null, new Key(ns, set, key));
                    log("Req exc time: " + (System.currentTimeMillis() - before) + "ms");
                } catch (Exception e) {
                    log("BOOM! " + e.getMessage());
                }
            }
        }

output:

Req: 1 -- 1467987976901
Req exc time: 25ms
Req: 2 -- 1467987976926
Req exc time: 6ms
Req: 3 -- 1467987976933
BOOM! Client timeout: timeout=300 iterations=1 failedNodes=0 failedConns=0 lastNode=BB9000100000001 10.0.0.11:3000
Req: 4 -- 1467987977234
BOOM! Client timeout: timeout=300 iterations=1 failedNodes=0 failedConns=0 lastNode=BB9000200000002 10.0.0.12:3000
Req: 5 -- 1467987977535
Req exc time: 6ms
Req: 6 -- 1467987977541
Req exc time: 6ms
Req: 7 -- 1467987977547
BOOM! Client timeout: timeout=300 iterations=1 failedNodes=0 failedConns=0 lastNode=BB9000100000001 10.0.0.11:3000
Req: 8 -- 1467987977848
BOOM! Client timeout: timeout=300 iterations=1 failedNodes=0 failedConns=0 lastNode=BB9000200000002 10.0.0.12:3000
Req: 9 -- 1467987978149
Req exc time: 5ms
Req: 10 -- 1467987978154
Req exc time: 7ms
Req: 11 -- 1467987978161
BOOM! Client timeout: timeout=300 iterations=1 failedNodes=0 failedConns=0 lastNode=BB9000100000001 10.0.0.11:3000
Req: 12 -- 1467987978462
BOOM! Client timeout: timeout=300 iterations=1 failedNodes=0 failedConns=0 lastNode=BB9000200000002 10.0.0.12:3000
[...]

Am I missing something? Is there the need to release some resources? The execution times of the first two requests would be good…

Thanks, Giacomo


#2

Although “maxConnsPerNode=1” should technically work for your test case, I suggest using the default “maxConnsPerNode=300” instead and see if that makes a difference.


#3

Hi Brian,

thanks for you reply. I tried with many different values for maxConnsPerNode but nothing changed. I noticed that in every case no more than two or three connections were made towards the servers, and the connections kept changing, so I suspected a network issue. I tried to run the client directly on one of the servers, and got no problems even at “maxConnsPerNode = 1”, so I suppose it’s a networking issue between the host I was using as client and the servers, I will investigate this further.

Thanks, Giacomo