The issue about use java client benchmark

Benchmarks Aerospike Server Benchmarks
1

I run following java client benchmark, I encounter “peers-clear-std response is empty” issue. I know sometimes, the timeout is normal on benchmark test, but it never throw any exception, just log timeout. I suspect there is any issue between my cluster. Maybe connection or others.

./run_benchmarks -h 10.16.100.4 -p 3000 -n production -k 1000 -S 1 -o B:120000 -b 3 -w RU,0 -z 20 -T 10000

2017-06-06 09:36:50.961 write(tps=119 timeouts=0 errors=0) read(tps=0 timeouts=0 errors=0) total(tps=119 timeouts=0 errors=0)
2017-06-06 09:36:51.513 WARN Thread tend Node BB9D1625C3EF262 10.16.100.6 3000 refresh failed: java.net.SocketTimeoutException: Read timed out
2017-06-06 09:36:51.515 WARN Thread tend Node BB953EBC8D8CC22 10.16.100.4 3000 refresh failed: com.aerospike.client.AerospikeException$Parse: Error Code -2: peers-clear-std response is empty
        at com.aerospike.client.cluster.PeerParser.<init>(PeerParser.java:46)
        at com.aerospike.client.cluster.Node.refreshPeers(Node.java:311)
        at com.aerospike.client.cluster.Cluster.tend(Cluster.java:363)
        at com.aerospike.client.cluster.Cluster.run(Cluster.java:317)
        at java.lang.Thread.run(Thread.java:748)

2017-06-06 09:36:51.516 WARN Thread tend Node BB9AAC121CE6DB2 10.16.100.8 3000 refresh failed: com.aerospike.client.AerospikeException$Parse: Error Code -2: peers-clear-std response is empty
        at com.aerospike.client.cluster.PeerParser.<init>(PeerParser.java:46)
        at com.aerospike.client.cluster.Node.refreshPeers(Node.java:311)
        at com.aerospike.client.cluster.Cluster.tend(Cluster.java:363)
        at com.aerospike.client.cluster.Cluster.run(Cluster.java:317)
        at java.lang.Thread.run(Thread.java:748)
2

Sounds like its getting an empty response of nodes that should have a list of nodes in the cluster. Can you supply versions?

3

Aerospike CE 3.9.1

4

Version of benchmark client?

5

the latest one, it seems 4.0.1.

6

And the client version you have installed?

7

java client version is 4.0.1 commit 859ad8f01208d56cc4016ffc11036c7bd7d05f97 Author: Brian Nichols bnichols@aerospike.com Date: Tue May 30 15:53:41 2017 -0700

Update version 4.0.1
8

The “peers-clear-std response is empty” is a benign bug. Here is what is happening.

  1. The cluster tend request to two nodes fails. The expected “refresh failed” log message is displayed for these nodes.

  2. The client mistakenly assumes that the peers protocol is enabled after this failure and requests peers from the server.

  3. Your server version 3.9.1 does not support the peers protocol and returns an error.

We will modify the java client to only request peers on servers that support it (server versions >= 3.10.x).

9

Thanks, if currently I cannot upgrade server, which version benchmark do you recommend？

10

Your current java client will work, just ignore the “peers” error message. The peers error message will only occur when the cluster tend can’t reach nodes.

You need to resolve why the client can’t reliably reach 10.16.100.6 within the default 1 second timeout. That is a different issue altogether.

11

does mesh mode support 1 node disconnect and the cluster work fine？ it seems if one node disconnect to the cluster, many heartbeat warning message in other node and in the client also report it.