java.net.SocketException: Connection reset, Error -8 on large set query

nishant.nitjsr · May 20, 2020, 1:33pm

Hi,

I am getting Connection reset error sometimes while querying a large set. I am using secondary index value (always 1) as a filter. This query should return approx ~1.5M record (all records in the set). I am using 4.5.0.9 server version (8 nodes) and 4.4.9 java client version (~30 clients query every hour).

2020-05-20 12:42:50.739 [ERROR] [main] : Exception in reloading value cache
com.aerospike.client.AerospikeException$Connection: Error -8,1,210000,0,5,BB912D1B96B1FAC 10.15.20.75 3000: java.net.SocketException: Connection reset
  at com.aerospike.client.command.SyncCommand.executeCommand(SyncCommand.java:168) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:75) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.command.MultiCommand.executeAndValidate(MultiCommand.java:98) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.query.QueryExecutor$QueryThread.run(QueryExecutor.java:134) ~[aerospike-client-4.4.9.jar:?]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
  at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.net.SocketException: Connection reset
  at java.net.SocketInputStream.read(SocketInputStream.java:186) ~[?:?]
  at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
  at com.aerospike.client.cluster.Connection.readFully(Connection.java:273) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.command.MultiCommand.parseResult(MultiCommand.java:153) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.command.SyncCommand.executeCommand(SyncCommand.java:114) ~[aerospike-client-4.4.9.jar:?]
  ... 6 more
2020-05-20 12:42:50.740 [ERROR] [main] : Exception:
com.aerospike.client.AerospikeException$Connection: Error -8,1,210000,0,5,BB912D1B96B1FAC 10.15.20.75 3000: java.net.SocketException: Connection reset
  at com.aerospike.client.command.SyncCommand.executeCommand(SyncCommand.java:168) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:75) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.command.MultiCommand.executeAndValidate(MultiCommand.java:98) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.query.QueryExecutor$QueryThread.run(QueryExecutor.java:134) ~[aerospike-client-4.4.9.jar:?]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
  at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.net.SocketException: Connection reset
  at java.net.SocketInputStream.read(SocketInputStream.java:186) ~[?:?]
  at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
  at com.aerospike.client.cluster.Connection.readFully(Connection.java:273) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.command.MultiCommand.parseResult(MultiCommand.java:153) ~[aerospike-client-4.4.9.jar:?]
  at com.aerospike.client.command.SyncCommand.executeCommand(SyncCommand.java:114) ~[aerospike-client-4.4.9.jar:?]
  ... 6 more

client policy:

  public static ClientPolicy getClientPolicy(int eventLoopCount) {
    EventPolicy eventPolicy = new EventPolicy();
    eventPolicy.minTimeout = 5;
    EventLoopGroup group = new EpollEventLoopGroup(eventLoopCount);

    EventLoops eventLoops = new NettyEventLoops(eventPolicy, group);
    ClientPolicy clientPolicy = new ClientPolicy();
    clientPolicy.eventLoops = eventLoops;
    clientPolicy.maxConnsPerNode = 9000;
    
    BatchPolicy batchPolicy = new BatchPolicy();
    batchPolicy.socketTimeout = 10; //10ms
    batchPolicy.totalTimeout = 30; // 30ms
    batchPolicy.maxRetries = 1; // Retry max 1, so total attempt = 2

    // Default for write/query/scan: 0 (no retries)
    QueryPolicy queryPolicy = new QueryPolicy();
    queryPolicy.compress = true;
    queryPolicy.recordQueueSize = 10000;
    queryPolicy.socketTimeout = 210000; //210s

    clientPolicy.queryPolicyDefault = queryPolicy;
    clientPolicy.batchPolicyDefault = batchPolicy;
    
    return clientPolicy;
  }

How can I solve it?

Brian · May 28, 2020, 12:16am

Does the Connection reset error occur after 210 seconds?

Query is the only command where the server closes the socket on timeout or other socket error. All other commands rely on the client to close the socket.

nishant.nitjsr · May 28, 2020, 5:19am

I have seen it before 210 sec. See the time diff in the log from start to error.

2020-05-27 16:38:05.621 [INFO ] [main] : cache reloading started
2020-05-27 16:38:19.126 [ERROR] [main] : Exception in reloading value cache
com.aerospike.client.AerospikeException$Connection: Error -8,1,210000,0,5,BB912D1B96B1FAC 10.15.20.75 3000: java.net.SocketException: Connection reset
        at com.aerospike.client.command.SyncCommand.executeCommand(SyncCommand.java:168) ~[aerospike-client-4.4.9.jar:?]
        at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:75) ~[aerospike-client-4.4.9.jar:?]
        at com.aerospike.client.command.MultiCommand.executeAndValidate(MultiCommand.java:98) ~[aerospike-client-4.4.9.jar:?]
        at com.aerospike.client.query.QueryExecutor$QueryThread.run(QueryExecutor.java:134) ~[aerospike-client-4.4.9.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:186) ~[?:?]
        at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
        at com.aerospike.client.cluster.Connection.readFully(Connection.java:273) ~[aerospike-client-4.4.9.jar:?]
        at com.aerospike.client.command.MultiCommand.parseResult(MultiCommand.java:153) ~[aerospike-client-4.4.9.jar:?]
        at com.aerospike.client.command.SyncCommand.executeCommand(SyncCommand.java:114) ~[aerospike-client-4.4.9.jar:?]
        ... 6 more

In general, it takes 15-20 sec to finish reloading.

2020-05-27 16:43:35.902 [INFO ] [main] : cache reloading started
2020-05-27 16:43:55.253 [INFO ] [main] : cache reload finished

I have observed that it happens mostly during application restart. We reload this cache on application startup and then internally every hour via scheduled timer thread.

Brian · May 28, 2020, 8:00pm

The server should log a message before it closes the query connection. Check server logs for a more detailed explanation.

nishant.nitjsr · May 29, 2020, 8:27am

I didn’t find anything in that server (Node: bb912d1b96b1fac) log:

aerospike conf:

logging {
  # Log file must be an absolute path.
  file /var/log/aerospike/aerospike.log {
    context any info
  }
}

Node: bb912d1b96b1fac aerospike.log during that time window -

[aerosdbSL@aeros8 ~]$ head /var/log/aerospike/aerospike.log-20200528
May 27 2020 03:26:08 GMT: INFO (info): (ticker.c:160) NODE-ID bb912d1b96b1fac CLUSTER-SIZE 8
May 27 2020 03:26:08 GMT: INFO (info): (ticker.c:233)    cluster-clock: skew-ms 272395 outliers (BB91AC9B96B1FAC)
May 27 2020 03:26:08 GMT: INFO (info): (ticker.c:265)    system-memory: free-kbytes 58318216 free-pct 14 heap-kbytes (232298889,316802732,336545792) heap-efficiency-pct 69.0
May 27 2020 03:26:08 GMT: INFO (info): (ticker.c:278)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
May 27 2020 03:26:08 GMT: INFO (info): (ticker.c:299)    fds: proto (1443,37030052,37028609) heartbeat (7,88,81) fabric (168,579,411)
May 27 2020 03:26:08 GMT: INFO (info): (ticker.c:307)    heartbeat-received: self 2 foreign 655847392
May 27 2020 03:26:08 GMT: INFO (info): (ticker.c:337)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (82954,84842)
May 27 2020 03:26:08 GMT: INFO (info): (ticker.c:360)    early-fail: demarshal 0 tsvc-client 1 tsvc-batch-sub 0 tsvc-udf-sub 0
May 27 2020 03:26:08 GMT: INFO (info): (ticker.c:376)    batch-index: batches (450075463314,1,18260)
May 27 2020 03:26:08 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (450075463315 total) msec
[aerosdbSL@aeros8 ~]$ grep -v INFO /var/log/aerospike/aerospike.log-20200528

complete log of that period -

May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:160) NODE-ID bb912d1b96b1fac CLUSTER-SIZE 8
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:233)    cluster-clock: skew-ms 272505 outliers (BB91AC9B96B1FAC)
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:265)    system-memory: free-kbytes 58279824 free-pct 14 heap-kbytes (232431696,316830320,336568320) heap-efficiency-pct 69.1
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:278)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:299)    fds: proto (1796,37360161,37358365) heartbeat (7,88,81) fabric (168,579,411)
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:307)    heartbeat-received: self 2 foreign 658063482
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:337)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (121819,126841)
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:360)    early-fail: demarshal 0 tsvc-client 1 tsvc-batch-sub 0 tsvc-udf-sub 0
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:376)    batch-index: batches (451765455222,1,18260)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (451765455224 total) msec
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (00: 451764809183) (01: 0000378409) (02: 0000076180) (03: 0000082075)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (04: 0000020587) (05: 0000001756) (06: 0000003975) (07: 0000008607)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (08: 0000015135) (09: 0000026021) (10: 0000032724) (11: 0000000572)
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:447) {test} migrations: complete
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:389) {production} objects: all 1172484847 master 552160872 prole 620323975 non-replica 0
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:447) {production} migrations: complete
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:465) {production} memory-usage: total-bytes 183383272526 index-bytes 75039030208 sindex-bytes 618011105 data-bytes 107726231213 used-pct 48.80
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:517) {production} device-usage: used-bytes 148965586736 avail-pct 78
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:585) {production} client: tsvc (0,6492) proxy (738,0,0) read (212,0,0,13) write (5368578302,288773,2807) delete (0,0,0,0) udf (844189584,0,466) lang (933984,759458963,83797101,0)
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:635) {production} batch-sub: tsvc (0,0) proxy (109439,0,18260) read (299201673160,0,0,153731502715)
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:662) {production} scan: basic (117,55,0) aggr (0,0,0) udf-bg (0,0,0)
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:686) {production} query: basic (36108724,75) aggr (0,0) udf-bg (0,0)
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:750) {production} retransmits: migration 104923451 client-read 0 client-write (2132,1000) client-delete (0,0) client-udf (7,301) batch-sub 0 udf-sub (0,0)
May 27 2020 16:37:53 GMT: INFO (info): (ticker.c:785) {production} special-errors: key-busy 288765 record-too-big 0
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-read (225 total) msec
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:266)  (00: 0000000225)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-write (5368867075 total) msec
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (00: 5368425657) (01: 0000283186) (02: 0000038297) (03: 0000030593)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (04: 0000025359) (05: 0000019319) (06: 0000006576) (07: 0000006226)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (08: 0000017353) (09: 0000005961) (10: 0000008488) (11: 0000000060)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-udf (844189584 total) msec
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (00: 0843688785) (01: 0000133667) (02: 0000065963) (03: 0000113724)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (04: 0000156687) (05: 0000027744) (06: 0000001222) (07: 0000000537)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (08: 0000000793) (09: 0000000214) (10: 0000000244) (11: 0000000004)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query (36108799 total) msec
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (00: 0000111965) (01: 0034187666) (02: 0000011702) (03: 0000011137)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (04: 0000025457) (05: 0000305030) (06: 0000548641) (07: 0000372910)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (08: 0000316532) (09: 0000169789) (10: 0000009347) (11: 0000011919)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (12: 0000002538) (13: 0000023658) (14: 0000000496) (15: 0000000011)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:266)  (16: 0000000001)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query-rec-count (36020149 total) count
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (01: 0000000009) (02: 0000000043) (03: 0000000316) (04: 0000001646)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (05: 0000001596) (06: 0000004547) (07: 0033731541) (08: 0000678242)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (09: 0000022823) (10: 0000000052) (11: 0000000046) (12: 0000000130)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:257)  (13: 0000000301) (14: 0000955998) (15: 0000000161) (16: 0000573871)
May 27 2020 16:37:53 GMT: INFO (info): (hist.c:266)  (17: 0000000019) (18: 0000048808)
May 27 2020 16:37:54 GMT: INFO (nsup): (thr_sindex.c:521) {production} sindex-gc: Processed: 19118504, found:0, deleted: 0: Total time: 382994 ms
May 27 2020 16:37:55 GMT: INFO (nsup): (thr_sindex.c:490) {production} sindex-gc start
May 27 2020 16:37:58 GMT: INFO (drv_ssd): (drv_ssd.c:2185) {production} /opt/aerospike/data/production.data: used-bytes 148965623040 free-wblocks 804221 write-q 0 write (3362331,0.2) defrag-q 0 defrag-read (3153634,0.3) defrag-write (1509174,0.2)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:160) NODE-ID bb912d1b96b1fac CLUSTER-SIZE 8
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:233)    cluster-clock: skew-ms 272506 outliers (BB91AC9B96B1FAC)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:265)    system-memory: free-kbytes 58279868 free-pct 14 heap-kbytes (232431072,316830244,336568320) heap-efficiency-pct 69.1
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:278)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:299)    fds: proto (1813,37360288,37358475) heartbeat (7,88,81) fabric (168,579,411)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:307)    heartbeat-received: self 2 foreign 658063949
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:337)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (84857,87391)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:360)    early-fail: demarshal 0 tsvc-client 1 tsvc-batch-sub 0 tsvc-udf-sub 0
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:376)    batch-index: batches (451765853390,1,18260)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (451765853392 total) msec
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (00: 451765207350) (01: 0000378410) (02: 0000076180) (03: 0000082075)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (04: 0000020587) (05: 0000001756) (06: 0000003975) (07: 0000008607)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (08: 0000015135) (09: 0000026021) (10: 0000032724) (11: 0000000572)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:447) {test} migrations: complete
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:389) {production} objects: all 1172485221 master 552160876 prole 620324345 non-replica 0
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:447) {production} migrations: complete
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:465) {production} memory-usage: total-bytes 183383341625 index-bytes 75039054144 sindex-bytes 618011105 data-bytes 107726276376 used-pct 48.80
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:517) {production} device-usage: used-bytes 148965649696 avail-pct 78
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:585) {production} client: tsvc (0,6492) proxy (738,0,0) read (212,0,0,13) write (5368580781,288773,2807) delete (0,0,0,0) udf (844189868,0,466) lang (933984,759459247,83797101,0)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:635) {production} batch-sub: tsvc (0,0) proxy (109439,0,18260) read (299201928660,0,0,153731646849)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:662) {production} scan: basic (117,55,0) aggr (0,0,0) udf-bg (0,0,0)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:686) {production} query: basic (36108789,75) aggr (0,0) udf-bg (0,0)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:750) {production} retransmits: migration 104923451 client-read 0 client-write (2132,1000) client-delete (0,0) client-udf (7,301) batch-sub 0 udf-sub (0,0)
May 27 2020 16:38:03 GMT: INFO (info): (ticker.c:785) {production} special-errors: key-busy 288765 record-too-big 0
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-read (225 total) msec
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:266)  (00: 0000000225)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-write (5368869554 total) msec
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (00: 5368428136) (01: 0000283186) (02: 0000038297) (03: 0000030593)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (04: 0000025359) (05: 0000019319) (06: 0000006576) (07: 0000006226)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (08: 0000017353) (09: 0000005961) (10: 0000008488) (11: 0000000060)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-udf (844189868 total) msec
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (00: 0843689069) (01: 0000133667) (02: 0000065963) (03: 0000113724)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (04: 0000156687) (05: 0000027744) (06: 0000001222) (07: 0000000537)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (08: 0000000793) (09: 0000000214) (10: 0000000244) (11: 0000000004)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query (36108864 total) msec
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (00: 0000111965) (01: 0034187725) (02: 0000011702) (03: 0000011137)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (04: 0000025457) (05: 0000305030) (06: 0000548641) (07: 0000372912)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (08: 0000316532) (09: 0000169793) (10: 0000009347) (11: 0000011919)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (12: 0000002538) (13: 0000023658) (14: 0000000496) (15: 0000000011)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:266)  (16: 0000000001)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query-rec-count (36020214 total) count
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (01: 0000000009) (02: 0000000043) (03: 0000000316) (04: 0000001646)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (05: 0000001596) (06: 0000004547) (07: 0033731600) (08: 0000678242)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (09: 0000022823) (10: 0000000052) (11: 0000000046) (12: 0000000130)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:257)  (13: 0000000301) (14: 0000956000) (15: 0000000161) (16: 0000573875)
May 27 2020 16:38:03 GMT: INFO (info): (hist.c:266)  (17: 0000000019) (18: 0000048808)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:160) NODE-ID bb912d1b96b1fac CLUSTER-SIZE 8
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:233)    cluster-clock: skew-ms 272506 outliers (BB91AC9B96B1FAC)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:265)    system-memory: free-kbytes 58279856 free-pct 14 heap-kbytes (232431100,316830136,336568320) heap-efficiency-pct 69.1
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:278)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:299)    fds: proto (1822,37360362,37358540) heartbeat (7,88,81) fabric (168,579,411)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:307)    heartbeat-received: self 2 foreign 658064413
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:337)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (122473,128522)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:360)    early-fail: demarshal 0 tsvc-client 1 tsvc-batch-sub 0 tsvc-udf-sub 0
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:376)    batch-index: batches (451766249276,1,18260)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (451766249277 total) msec
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (00: 451765603235) (01: 0000378410) (02: 0000076180) (03: 0000082075)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (04: 0000020587) (05: 0000001756) (06: 0000003975) (07: 0000008607)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (08: 0000015135) (09: 0000026021) (10: 0000032724) (11: 0000000572)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:447) {test} migrations: complete
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:389) {production} objects: all 1172484626 master 552160457 prole 620324168 non-replica 0
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:447) {production} migrations: complete
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:465) {production} memory-usage: total-bytes 183383228535 index-bytes 75039016064 sindex-bytes 618011105 data-bytes 107726201366 used-pct 48.80
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:517) {production} device-usage: used-bytes 148965539168 avail-pct 78
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:585) {production} client: tsvc (0,6492) proxy (738,0,0) read (212,0,0,13) write (5368583005,288773,2807) delete (0,0,0,0) udf (844191715,0,466) lang (933984,759460681,83797514,0)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:635) {production} batch-sub: tsvc (0,0) proxy (109439,0,18260) read (299202182650,0,0,153731790346)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:662) {production} scan: basic (117,55,0) aggr (0,0,0) udf-bg (0,0,0)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:686) {production} query: basic (36108852,75) aggr (0,0) udf-bg (0,0)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:750) {production} retransmits: migration 104923451 client-read 0 client-write (2132,1000) client-delete (0,0) client-udf (7,301) batch-sub 0 udf-sub (0,0)
May 27 2020 16:38:13 GMT: INFO (info): (ticker.c:785) {production} special-errors: key-busy 288765 record-too-big 0
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:240) histogram dump: {production}-read (225 total) msec
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:266)  (00: 0000000225)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:240) histogram dump: {production}-write (5368871778 total) msec
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (00: 5368430360) (01: 0000283186) (02: 0000038297) (03: 0000030593)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (04: 0000025359) (05: 0000019319) (06: 0000006576) (07: 0000006226)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (08: 0000017353) (09: 0000005961) (10: 0000008488) (11: 0000000060)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:240) histogram dump: {production}-udf (844191715 total) msec
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (00: 0843690916) (01: 0000133667) (02: 0000065963) (03: 0000113724)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (04: 0000156687) (05: 0000027744) (06: 0000001222) (07: 0000000537)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (08: 0000000793) (09: 0000000214) (10: 0000000244) (11: 0000000004)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query (36108927 total) msec
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (00: 0000111965) (01: 0034187783) (02: 0000011702) (03: 0000011137)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (04: 0000025457) (05: 0000305030) (06: 0000548641) (07: 0000372916)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (08: 0000316532) (09: 0000169793) (10: 0000009347) (11: 0000011919)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (12: 0000002539) (13: 0000023658) (14: 0000000496) (15: 0000000011)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:266)  (16: 0000000001)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query-rec-count (36020277 total) count
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (01: 0000000009) (02: 0000000043) (03: 0000000316) (04: 0000001646)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (05: 0000001596) (06: 0000004547) (07: 0033731658) (08: 0000678242)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (09: 0000022823) (10: 0000000052) (11: 0000000046) (12: 0000000130)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:257)  (13: 0000000301) (14: 0000956004) (15: 0000000161) (16: 0000573875)
May 27 2020 16:38:13 GMT: INFO (info): (hist.c:266)  (17: 0000000019) (18: 0000048809)
May 27 2020 16:38:18 GMT: INFO (drv_ssd): (drv_ssd.c:2185) {production} /opt/aerospike/data/production.data: used-bytes 148965571600 free-wblocks 804223 write-q 0 write (3362336,0.2) defrag-q 0 defrag-read (3153640,0.3) defrag-write (1509177,0.2)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:160) NODE-ID bb912d1b96b1fac CLUSTER-SIZE 8
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:233)    cluster-clock: skew-ms 272506 outliers (BB91AC9B96B1FAC)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:265)    system-memory: free-kbytes 58279844 free-pct 14 heap-kbytes (232430912,316829936,336568320) heap-efficiency-pct 69.1
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:278)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:299)    fds: proto (1830,37360428,37358598) heartbeat (7,88,81) fabric (168,579,411)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:307)    heartbeat-received: self 2 foreign 658064882
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:337)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (117806,124368)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:360)    early-fail: demarshal 0 tsvc-client 1 tsvc-batch-sub 0 tsvc-udf-sub 0
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:376)    batch-index: batches (451766648510,1,18260)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (451766648512 total) msec
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (00: 451766002469) (01: 0000378411) (02: 0000076180) (03: 0000082075)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (04: 0000020587) (05: 0000001756) (06: 0000003975) (07: 0000008607)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (08: 0000015135) (09: 0000026021) (10: 0000032724) (11: 0000000572)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:447) {test} migrations: complete
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:389) {production} objects: all 1172484071 master 552160104 prole 620323967 non-replica 0
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:447) {production} migrations: complete
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:465) {production} memory-usage: total-bytes 183383119756 index-bytes 75038980544 sindex-bytes 618011105 data-bytes 107726128107 used-pct 48.80
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:517) {production} device-usage: used-bytes 148965432256 avail-pct 78
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:585) {production} client: tsvc (0,6492) proxy (738,0,0) read (212,0,0,13) write (5368585050,288773,2807) delete (0,0,0,0) udf (844193536,0,466) lang (933984,759462122,83797894,0)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:635) {production} batch-sub: tsvc (0,0) proxy (109439,0,18260) read (299202439059,0,0,153731934668)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:662) {production} scan: basic (117,55,0) aggr (0,0,0) udf-bg (0,0,0)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:686) {production} query: basic (36108914,75) aggr (0,0) udf-bg (0,0)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:750) {production} retransmits: migration 104923451 client-read 0 client-write (2132,1000) client-delete (0,0) client-udf (7,301) batch-sub 0 udf-sub (0,0)
May 27 2020 16:38:23 GMT: INFO (info): (ticker.c:785) {production} special-errors: key-busy 288765 record-too-big 0
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:240) histogram dump: {production}-read (225 total) msec
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:266)  (00: 0000000225)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:240) histogram dump: {production}-write (5368873823 total) msec
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (00: 5368432405) (01: 0000283186) (02: 0000038297) (03: 0000030593)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (04: 0000025359) (05: 0000019319) (06: 0000006576) (07: 0000006226)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (08: 0000017353) (09: 0000005961) (10: 0000008488) (11: 0000000060)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:240) histogram dump: {production}-udf (844193536 total) msec
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (00: 0843692737) (01: 0000133667) (02: 0000065963) (03: 0000113724)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (04: 0000156687) (05: 0000027744) (06: 0000001222) (07: 0000000537)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (08: 0000000793) (09: 0000000214) (10: 0000000244) (11: 0000000004)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query (36108989 total) msec
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (00: 0000111965) (01: 0034187842) (02: 0000011702) (03: 0000011137)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (04: 0000025457) (05: 0000305030) (06: 0000548641) (07: 0000372918)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (08: 0000316532) (09: 0000169794) (10: 0000009347) (11: 0000011919)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (12: 0000002539) (13: 0000023658) (14: 0000000496) (15: 0000000011)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:266)  (16: 0000000001)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query-rec-count (36020339 total) count
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (01: 0000000009) (02: 0000000043) (03: 0000000316) (04: 0000001646)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (05: 0000001596) (06: 0000004547) (07: 0033731717) (08: 0000678242)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (09: 0000022823) (10: 0000000052) (11: 0000000046) (12: 0000000130)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:257)  (13: 0000000301) (14: 0000956006) (15: 0000000161) (16: 0000573876)
May 27 2020 16:38:23 GMT: INFO (info): (hist.c:266)  (17: 0000000019) (18: 0000048809)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:160) NODE-ID bb912d1b96b1fac CLUSTER-SIZE 8
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:233)    cluster-clock: skew-ms 272506 outliers (BB91AC9B96B1FAC)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:265)    system-memory: free-kbytes 58279792 free-pct 14 heap-kbytes (232430996,316829908,336568320) heap-efficiency-pct 69.1
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:278)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:299)    fds: proto (1879,37360540,37358661) heartbeat (7,88,81) fabric (168,579,411)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:307)    heartbeat-received: self 2 foreign 658065348
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:337)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (69251,71077)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:360)    early-fail: demarshal 0 tsvc-client 1 tsvc-batch-sub 0 tsvc-udf-sub 0
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:376)    batch-index: batches (451767045320,1,18260)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (451767045321 total) msec
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (00: 451766399278) (01: 0000378411) (02: 0000076180) (03: 0000082075)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (04: 0000020587) (05: 0000001756) (06: 0000003975) (07: 0000008607)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (08: 0000015135) (09: 0000026021) (10: 0000032724) (11: 0000000572)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:447) {test} migrations: complete
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:389) {production} objects: all 1172484339 master 552160138 prole 620324201 non-replica 0
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:447) {production} migrations: complete
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:465) {production} memory-usage: total-bytes 183383171778 index-bytes 75038997696 sindex-bytes 618011105 data-bytes 107726162977 used-pct 48.80
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:517) {production} device-usage: used-bytes 148965480384 avail-pct 78
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:585) {production} client: tsvc (0,6492) proxy (738,0,0) read (212,0,0,13) write (5368586939,288773,2807) delete (0,0,0,0) udf (844193849,0,466) lang (933984,759462435,83797894,0)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:635) {production} batch-sub: tsvc (0,0) proxy (109439,0,18260) read (299202692668,0,0,153732079504)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:662) {production} scan: basic (117,55,0) aggr (0,0,0) udf-bg (0,0,0)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:686) {production} query: basic (36108973,75) aggr (0,0) udf-bg (0,0)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:750) {production} retransmits: migration 104923451 client-read 0 client-write (2132,1000) client-delete (0,0) client-udf (7,301) batch-sub 0 udf-sub (0,0)
May 27 2020 16:38:33 GMT: INFO (info): (ticker.c:785) {production} special-errors: key-busy 288765 record-too-big 0
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:240) histogram dump: {production}-read (225 total) msec
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:266)  (00: 0000000225)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:266)  (00: 0000000225)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:240) histogram dump: {production}-write (5368875712 total) msec
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (00: 5368434294) (01: 0000283186) (02: 0000038297) (03: 0000030593)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (04: 0000025359) (05: 0000019319) (06: 0000006576) (07: 0000006226)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (08: 0000017353) (09: 0000005961) (10: 0000008488) (11: 0000000060)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:240) histogram dump: {production}-udf (844193849 total) msec
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (00: 0843693050) (01: 0000133667) (02: 0000065963) (03: 0000113724)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (04: 0000156687) (05: 0000027744) (06: 0000001222) (07: 0000000537)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (08: 0000000793) (09: 0000000214) (10: 0000000244) (11: 0000000004)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query (36109048 total) msec
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (00: 0000111965) (01: 0034187900) (02: 0000011702) (03: 0000011137)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (04: 0000025457) (05: 0000305030) (06: 0000548641) (07: 0000372919)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (08: 0000316532) (09: 0000169794) (10: 0000009347) (11: 0000011919)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (12: 0000002539) (13: 0000023658) (14: 0000000496) (15: 0000000011)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:266)  (16: 0000000001)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query-rec-count (36020398 total) count
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (01: 0000000009) (02: 0000000043) (03: 0000000316) (04: 0000001646)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (05: 0000001596) (06: 0000004547) (07: 0033731775) (08: 0000678242)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (09: 0000022823) (10: 0000000052) (11: 0000000046) (12: 0000000130)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:257)  (13: 0000000301) (14: 0000956007) (15: 0000000161) (16: 0000573876)
May 27 2020 16:38:33 GMT: INFO (info): (hist.c:266)  (17: 0000000019) (18: 0000048809)
May 27 2020 16:38:38 GMT: INFO (drv_ssd): (drv_ssd.c:2185) {production} /opt/aerospike/data/production.data: used-bytes 148965366848 free-wblocks 804221 write-q 0 write (3362340,0.2) defrag-q 0 defrag-read (3153644,0.2) defrag-write (1509179,0.1)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:160) NODE-ID bb912d1b96b1fac CLUSTER-SIZE 8
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:233)    cluster-clock: skew-ms 272506 outliers (BB91AC9B96B1FAC)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:265)    system-memory: free-kbytes 58279792 free-pct 14 heap-kbytes (232431565,316829856,336568320) heap-efficiency-pct 69.1
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:278)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:299)    fds: proto (1883,37360616,37358733) heartbeat (7,88,81) fabric (168,579,411)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:307)    heartbeat-received: self 2 foreign 658065813
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:337)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (123876,129498)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:360)    early-fail: demarshal 0 tsvc-client 1 tsvc-batch-sub 0 tsvc-udf-sub 0
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:376)    batch-index: batches (451767438979,1,18260)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (451767438980 total) msec
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (00: 451766792936) (01: 0000378412) (02: 0000076180) (03: 0000082075)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (04: 0000020587) (05: 0000001756) (06: 0000003975) (07: 0000008607)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (08: 0000015135) (09: 0000026021) (10: 0000032724) (11: 0000000572)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:447) {test} migrations: complete
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:389) {production} objects: all 1172483940 master 552159815 prole 620324125 non-replica 0
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:447) {production} migrations: complete
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:465) {production} memory-usage: total-bytes 183383085955 index-bytes 75038972160 sindex-bytes 618011105 data-bytes 107726102690 used-pct 48.80
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:517) {production} device-usage: used-bytes 148965392656 avail-pct 78
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:585) {production} client: tsvc (0,6492) proxy (738,0,0) read (212,0,0,13) write (5368589132,288773,2807) delete (0,0,0,0) udf (844195714,0,466) lang (933984,759463934,83798260,0)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:635) {production} batch-sub: tsvc (0,0) proxy (109439,0,18260) read (299202944498,0,0,153732222804)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:662) {production} scan: basic (117,55,0) aggr (0,0,0) udf-bg (0,0,0)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:686) {production} query: basic (36109034,75) aggr (0,0) udf-bg (0,0)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:750) {production} retransmits: migration 104923451 client-read 0 client-write (2132,1000) client-delete (0,0) client-udf (7,301) batch-sub 0 udf-sub (0,0)
May 27 2020 16:38:43 GMT: INFO (info): (ticker.c:785) {production} special-errors: key-busy 288765 record-too-big 0
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:240) histogram dump: {production}-read (225 total) msec
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:266)  (00: 0000000225)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:240) histogram dump: {production}-write (5368877905 total) msec
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (00: 5368436487) (01: 0000283186) (02: 0000038297) (03: 0000030593)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (04: 0000025359) (05: 0000019319) (06: 0000006576) (07: 0000006226)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (08: 0000017353) (09: 0000005961) (10: 0000008488) (11: 0000000060)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:240) histogram dump: {production}-udf (844195714 total) msec
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (00: 0843694915) (01: 0000133667) (02: 0000065963) (03: 0000113724)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (04: 0000156687) (05: 0000027744) (06: 0000001222) (07: 0000000537)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (08: 0000000793) (09: 0000000214) (10: 0000000244) (11: 0000000004)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query (36109109 total) msec
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (00: 0000111965) (01: 0034187957) (02: 0000011702) (03: 0000011137)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (04: 0000025457) (05: 0000305030) (06: 0000548641) (07: 0000372920)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (08: 0000316533) (09: 0000169796) (10: 0000009347) (11: 0000011919)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (12: 0000002539) (13: 0000023658) (14: 0000000496) (15: 0000000011)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:266)  (16: 0000000001)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query-rec-count (36020459 total) count
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (01: 0000000009) (02: 0000000043) (03: 0000000316) (04: 0000001646)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (05: 0000001596) (06: 0000004547) (07: 0033731832) (08: 0000678242)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (09: 0000022823) (10: 0000000052) (11: 0000000046) (12: 0000000130)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:257)  (13: 0000000301) (14: 0000956009) (15: 0000000161) (16: 0000573878)
May 27 2020 16:38:43 GMT: INFO (info): (hist.c:266)  (17: 0000000019) (18: 0000048809)
May 27 2020 16:38:50 GMT: INFO (nsup): (thr_nsup.c:857) {production} nsup-done: master-objects (552157506,540264793) expired (885758851,6320) evicted (0,0) evict-ttl 0 waits (0,0) total-ms 82865
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:160) NODE-ID bb912d1b96b1fac CLUSTER-SIZE 8
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:233)    cluster-clock: skew-ms 272506 outliers (BB91AC9B96B1FAC)
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:265)    system-memory: free-kbytes 58279792 free-pct 14 heap-kbytes (232430047,316829772,336568320) heap-efficiency-pct 69.1
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:278)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:299)    fds: proto (1843,37360687,37358844) heartbeat (7,88,81) fabric (168,579,411)
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:307)    heartbeat-received: self 2 foreign 658066282
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:337)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (114756,122968)
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:360)    early-fail: demarshal 0 tsvc-client 1 tsvc-batch-sub 0 tsvc-udf-sub 0
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:376)    batch-index: batches (451767828285,1,18260)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (451767828286 total) msec
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (00: 451767182242) (01: 0000378412) (02: 0000076180) (03: 0000082075)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (04: 0000020587) (05: 0000001756) (06: 0000003975) (07: 0000008607)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (08: 0000015135) (09: 0000026021) (10: 0000032724) (11: 0000000572)
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:447) {test} migrations: complete
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:389) {production} objects: all 1172483491 master 552159682 prole 620323809 non-replica 0
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:447) {production} migrations: complete
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:465) {production} memory-usage: total-bytes 183382987550 index-bytes 75038943424 sindex-bytes 618011105 data-bytes 107726033021 used-pct 48.80
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:517) {production} device-usage: used-bytes 148965293760 avail-pct 78
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:585) {production} client: tsvc (0,6492) proxy (738,0,0) read (212,0,0,13) write (5368591118,288773,2807) delete (0,0,0,0) udf (844197594,0,466) lang (933984,759465453,83798621,0)
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:635) {production} batch-sub: tsvc (0,0) proxy (109439,0,18260) read (299203194435,0,0,153732363723)
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:662) {production} scan: basic (117,55,0) aggr (0,0,0) udf-bg (0,0,0)
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:686) {production} query: basic (36109093,75) aggr (0,0) udf-bg (0,0)
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:750) {production} retransmits: migration 104923451 client-read 0 client-write (2132,1000) client-delete (0,0) client-udf (7,301) batch-sub 0 udf-sub (0,0)
May 27 2020 16:38:53 GMT: INFO (info): (ticker.c:785) {production} special-errors: key-busy 288765 record-too-big 0
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-read (225 total) msec
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:266)  (00: 0000000225)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-write (5368879891 total) msec
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (00: 5368438472) (01: 0000283186) (02: 0000038298) (03: 0000030593)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (04: 0000025359) (05: 0000019319) (06: 0000006576) (07: 0000006226)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (08: 0000017353) (09: 0000005961) (10: 0000008488) (11: 0000000060)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-udf (844197594 total) msec
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (00: 0843696794) (01: 0000133668) (02: 0000065963) (03: 0000113724)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (04: 0000156687) (05: 0000027744) (06: 0000001222) (07: 0000000537)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (08: 0000000793) (09: 0000000214) (10: 0000000244) (11: 0000000004)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query (36109168 total) msec
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (00: 0000111965) (01: 0034188015) (02: 0000011702) (03: 0000011137)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (04: 0000025457) (05: 0000305030) (06: 0000548641) (07: 0000372921)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (08: 0000316533) (09: 0000169796) (10: 0000009347) (11: 0000011919)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (12: 0000002539) (13: 0000023658) (14: 0000000496) (15: 0000000011)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:266)  (16: 0000000001)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query-rec-count (36020518 total) count
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (01: 0000000009) (02: 0000000043) (03: 0000000316) (04: 0000001646)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (05: 0000001596) (06: 0000004547) (07: 0033731890) (08: 0000678242)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (09: 0000022823) (10: 0000000052) (11: 0000000046) (12: 0000000130)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:257)  (13: 0000000301) (14: 0000956010) (15: 0000000161) (16: 0000573878)
May 27 2020 16:38:53 GMT: INFO (info): (hist.c:266)  (17: 0000000019) (18: 0000048809)
May 27 2020 16:38:58 GMT: INFO (drv_ssd): (drv_ssd.c:2185) {production} /opt/aerospike/data/production.data: used-bytes 148965352032 free-wblocks 804224 write-q 0 write (3362343,0.2) defrag-q 0 defrag-read (3153648,0.2) defrag-write (1509181,0.1)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:160) NODE-ID bb912d1b96b1fac CLUSTER-SIZE 8
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:233)    cluster-clock: skew-ms 272506 outliers (BB91AC9B96B1FAC)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:265)    system-memory: free-kbytes 58279788 free-pct 14 heap-kbytes (232430310,316829868,336568320) heap-efficiency-pct 69.1
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:278)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:299)    fds: proto (1746,37360774,37359028) heartbeat (7,88,81) fabric (168,579,411)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:307)    heartbeat-received: self 2 foreign 658066749
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:337)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (69627,80099)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:360)    early-fail: demarshal 0 tsvc-client 1 tsvc-batch-sub 0 tsvc-udf-sub 0
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:376)    batch-index: batches (451768215848,1,18260)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (451768215849 total) msec
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (00: 451767569804) (01: 0000378413) (02: 0000076180) (03: 0000082075)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (04: 0000020587) (05: 0000001756) (06: 0000003975) (07: 0000008607)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (08: 0000015135) (09: 0000026021) (10: 0000032724) (11: 0000000572)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:447) {test} migrations: complete
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:389) {production} objects: all 1172484415 master 552160473 prole 620323942 non-replica 0
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:447) {production} migrations: complete
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:465) {production} memory-usage: total-bytes 183383123861 index-bytes 75039002560 sindex-bytes 618011105 data-bytes 107726110196 used-pct 48.80
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:517) {production} device-usage: used-bytes 148965408064 avail-pct 78
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:585) {production} client: tsvc (0,6492) proxy (738,0,0) read (212,0,0,13) write (5368593237,288773,2807) delete (0,0,0,0) udf (844197947,0,466) lang (933984,759465806,83798621,0)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:635) {production} batch-sub: tsvc (0,0) proxy (109439,0,18260) read (299203441767,0,0,153732505607)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:662) {production} scan: basic (117,55,0) aggr (0,0,0) udf-bg (0,0,0)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:686) {production} query: basic (36109150,75) aggr (0,0) udf-bg (0,0)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:750) {production} retransmits: migration 104923451 client-read 0 client-write (2132,1000) client-delete (0,0) client-udf (7,301) batch-sub 0 udf-sub (0,0)
May 27 2020 16:39:03 GMT: INFO (info): (ticker.c:785) {production} special-errors: key-busy 288765 record-too-big 0
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-read (225 total) msec
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:266)  (00: 0000000225)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-write (5368882010 total) msec
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (00: 5368440591) (01: 0000283186) (02: 0000038298) (03: 0000030593)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (04: 0000025359) (05: 0000019319) (06: 0000006576) (07: 0000006226)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (08: 0000017353) (09: 0000005961) (10: 0000008488) (11: 0000000060)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-udf (844197947 total) msec
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (00: 0843697147) (01: 0000133668) (02: 0000065963) (03: 0000113724)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (04: 0000156687) (05: 0000027744) (06: 0000001222) (07: 0000000537)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (08: 0000000793) (09: 0000000214) (10: 0000000244) (11: 0000000004)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query (36109225 total) msec
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (00: 0000111965) (01: 0034188071) (02: 0000011702) (03: 0000011137)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (04: 0000025457) (05: 0000305030) (06: 0000548641) (07: 0000372922)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (08: 0000316533) (09: 0000169796) (10: 0000009347) (11: 0000011919)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (12: 0000002539) (13: 0000023658) (14: 0000000496) (15: 0000000011)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:266)  (16: 0000000001)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:240) histogram dump: {production}-query-rec-count (36020575 total) count
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (01: 0000000009) (02: 0000000043) (03: 0000000316) (04: 0000001646)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (05: 0000001596) (06: 0000004547) (07: 0033731946) (08: 0000678242)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (09: 0000022823) (10: 0000000052) (11: 0000000046) (12: 0000000130)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:257)  (13: 0000000301) (14: 0000956011) (15: 0000000161) (16: 0000573878)
May 27 2020 16:39:03 GMT: INFO (info): (hist.c:266)  (17: 0000000019) (18: 0000048809)

Brian · June 1, 2020, 7:21pm

If there is no server message regarding an explicitly closed connection, I assume a connection error during the response or the query needed a new connection to be created and that failed.

Since the response involves 1.5M records, I suggest querying one node at a time (queryNode()) to reduce the rate of data sent to the client.

nishant.nitjsr · June 9, 2020, 12:28pm

I will try queryNode() method. I was wondering if i set maxConcurrentNodes to 1 in QueryPolicy wouldn’t it work same as queryNode()? It’s more like a workaround because queryNode() will require more affort to change.

Brian · June 9, 2020, 6:47pm

Yes, I forgot to mention that. Setting maxConcurrentNodes to 1 in QueryPolicy will also cause nodes to be queried sequentially.

system · June 16, 2020, 3:32pm

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
java.net.SocketException: Connection reset Java Client	4	3876	September 11, 2020
Query fails during migration Java Client query	5	264	December 18, 2023
Aerospike Python Client exception.ConnectionError: (-10, 'Socket read error: 11, Query & Indexing udf , error	0	1543	June 19, 2018
Immediately Connection reset exeption Java Client	4	1320	June 8, 2020
Possible reasons for error code -8 Java Client	8	4313	May 31, 2022

java.net.SocketException: Connection reset, Error -8 on large set query

Related Topics