This is when you crashed …
What did the server log have from this time when the node crashed? Do you see a “could not allocate” xxx “arena stage” … ? something similar?
i dont see could not allocate, but i can see that this node got disconnected from cluster, just before dying.
Jun 27 2022 16:45:32 GMT: WARNING (socket): (socket.c:900) (repeated:200) Error while connecting: 111 (Connection refused)
Jun 27 2022 16:45:32 GMT: WARNING (socket): (socket.c:900) (repeated:6) Error while connecting: 113 (No route to host)
Jun 27 2022 16:45:32 GMT: WARNING (socket): (socket.c:891) (repeated:74) Timeout while connecting
Jun 27 2022 16:45:32 GMT: INFO (drv_ssd): (drv_ssd.c:2185) {test2} /dev/vdb: used-bytes 60113717152 free-wblocks 2008754 write-q 0 write (548457137,7.9) defrag-q 0 defrag-read (548368522,8.8) defrag-write (265090539,4.3)
Jun 27 2022 16:45:42 GMT: INFO (hb): (hb.c:8066) node expired bb9b7cf210a0102
Jun 27 2022 16:45:42 GMT: INFO (fabric): (fabric.c:2486) fabric: node bb9b7cf210a0102 departed
Jun 27 2022 16:45:42 GMT: INFO (fabric): (fabric.c:923) fabric_node_disconnect(bb9b7cf210a0102)
Jun 27 2022 16:45:42 GMT: INFO (clustering): (clustering.c:7532) principal node bb9b7cf210a0102 departed - switching to orphan state
Jun 27 2022 16:45:42 GMT: INFO (clustering): (clustering.c:5646) moved self node to orphan state
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:160) NODE-ID bb95b91220a0102 CLUSTER-SIZE 8
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:238) cluster-clock: skew-ms 26
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:265) system-memory: free-kbytes 1605328 free-pct 1 heap-kbytes (72063950,88343508,116733952) heap-efficiency-pct 61.7
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:278) in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 3 proxy-hash 0 tree-gc-q 0
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:299) fds: proto (958,132777691,132776733) heartbeat (7,1163,1156) fabric (144,4560,4416)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:307) heartbeat-received: self 2 foreign 1896953250
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:337) fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (1565,1638)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:360) early-fail: demarshal 0 tsvc-client 15201 tsvc-batch-sub 0 tsvc-udf-sub 0
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:376) batch-index: batches (1922750314,233072,12)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (1922983386 total) msec
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (00: 1921169548) (01: 0000701391) (02: 0000471428) (03: 0000293080)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (04: 0000147671) (05: 0000070526) (06: 0000045002) (07: 0000030297)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (08: 0000030188) (09: 0000011800) (10: 0000006774) (11: 0000005558)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:266) (12: 0000000120) (13: 0000000003)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:389) {test2} objects: all 33797998 master 11352224 prole 22445774 non-replica 0
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:447) {test2} migrations: complete
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:465) {test2} memory-usage: total-bytes 62580877816 index-bytes 2163071872 sindex-bytes 2192812296 data-bytes 58224993648 used-pct 64.76
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:517) {test2} device-usage: used-bytes 60113717152 avail-pct 95
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:585) {test2} client: tsvc (0,3) proxy (555,0,32) read (39500823057,0,0,741801463) write (8614943971,19783,611) delete (12454417,0,1,17672660) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:635) {test2} batch-sub: tsvc (0,0) proxy (36,0,12) read (2992509667,0,0,3020503)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:662) {test2} scan: basic (90214,1620,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:686) {test2} query: basic (55282173195,12165) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:750) {test2} retransmits: migration 12208 client-read 0 client-write (0,59) client-delete (0,0) client-udf (0,0) batch-sub 0 udf-sub (0,0)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:785) {test2} special-errors: key-busy 63 record-too-big 17918
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-read (40242624520 total) msec
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (00: 40242595765) (01: 0000017936) (02: 0000007404) (03: 0000001805)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (04: 0000001185) (05: 0000000322) (06: 0000000063) (07: 0000000025)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:266) (08: 0000000002) (09: 0000000012) (10: 0000000001)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-write (8614963754 total) msec
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (00: 8308477699) (01: 0277638731) (02: 0020410818) (03: 0004604047)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (04: 0002235944) (05: 0001068044) (06: 0000287000) (07: 0000176658)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (08: 0000057846) (09: 0000004416) (10: 0000002535) (11: 0000000010)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:266) (12: 0000000006)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query (55282185360 total) msec
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (00: 55277799064) (01: 0002098031) (02: 0001068802) (03: 0000666988)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (04: 0000370901) (05: 0000141410) (06: 0000031785) (07: 0000005058)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (08: 0000001626) (09: 0000000292) (10: 0000000054) (11: 0000000025)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (12: 0000000013) (13: 0000000017) (14: 0000000653) (15: 0000000601)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (16: 0000000021) (17: 0000000003) (18: 0000000007) (19: 0000000001)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:266) (20: 0000000003) (21: 0000000005)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query-rec-count (10195923911 total) count
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (01: 7363952698) (02: 2057564691) (03: 0696637345) (04: 0063010429)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (05: 0008355226) (06: 0004445483) (07: 0001014600) (08: 0000535951)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (09: 0000303291) (10: 0000093732) (11: 0000007988) (12: 0000001101)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (13: 0000000028) (14: 0000000024) (15: 0000000011) (16: 0000000015)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:266) (17: 0000000013) (18: 0000001285)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:447) {test} migrations: complete
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:585) {test} client: tsvc (0,0) proxy (0,0,0) read (495312379,0,0,2647816) write (98664489,81060,15) delete (250301,0,0,55989) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:662) {test} scan: basic (5639,23,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:45:42 GMT: INFO (info): (ticker.c:686) {test} query: basic (349361864,0) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:240) histogram dump: {test}-read (497960195 total) msec
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (00: 0497960013) (01: 0000000109) (02: 0000000038) (03: 0000000011)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (04: 0000000004) (05: 0000000009) (06: 0000000002) (07: 0000000004)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:266) (08: 0000000003) (09: 0000000002)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:240) histogram dump: {test}-write (98745549 total) msec
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (00: 0098260376) (01: 0000440271) (02: 0000014651) (03: 0000006381)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (04: 0000007308) (05: 0000007824) (06: 0000004358) (07: 0000003445)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (08: 0000000864) (09: 0000000053) (10: 0000000017) (11: 0000000001)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query (349361864 total) msec
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (00: 0349361358) (01: 0000000317) (02: 0000000073) (03: 0000000055)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:266) (04: 0000000050) (05: 0000000011)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query-rec-count (50590271 total) count
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:257) (01: 0049917514) (02: 0000324346) (03: 0000162011) (04: 0000172119)
Jun 27 2022 16:45:42 GMT: INFO (info): (hist.c:266) (05: 0000014281)
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:42 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:42 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:42 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:42 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:42 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:42 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:42 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:900) (repeated:200) Error while connecting: 111 (Connection refused)
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:900) (repeated:6) Error while connecting: 113 (No route to host)
Jun 27 2022 16:45:42 GMT: WARNING (socket): (socket.c:891) (repeated:74) Timeout while connecting
Jun 27 2022 16:45:42 GMT: INFO (clustering): (clustering.c:5989) sent cluster join request to bb98fc7220a0102
Jun 27 2022 16:45:43 GMT: INFO (clustering): (clustering.c:5795) applied new cluster key bfdb9b4ca9f9
Jun 27 2022 16:45:43 GMT: INFO (clustering): (clustering.c:5797) applied new succession list bb98fc7220a0102 bb96d72220a0102 bb95b91220a0102 bb93f06200a0102 bb9125a200a0102 bb9080b200a0102 bb90739220a0102
Jun 27 2022 16:45:43 GMT: INFO (clustering): (clustering.c:5799) applied cluster size 7
Jun 27 2022 16:45:43 GMT: INFO (exchange): (exchange.c:2288) data exchange started with cluster key bfdb9b4ca9f9
Jun 27 2022 16:45:43 GMT: INFO (exchange): (exchange.c:3150) received commit command from principal node bb98fc7220a0102
Jun 27 2022 16:45:43 GMT: INFO (exchange): (exchange.c:3113) data exchange completed with cluster key bfdb9b4ca9f9
Jun 27 2022 16:45:43 GMT: INFO (partition): (partition_balance.c:956) {test2} replication factor is 3
Jun 27 2022 16:45:43 GMT: INFO (partition): (partition_balance.c:928) {test2} rebalanced: expected-migrations (268,230,536) fresh-partitions 0
Jun 27 2022 16:45:43 GMT: INFO (partition): (partition_balance.c:956) {test} replication factor is 3
Jun 27 2022 16:45:43 GMT: INFO (partition): (partition_balance.c:928) {test} rebalanced: expected-migrations (268,230,536) fresh-partitions 0
Jun 27 2022 16:45:43 GMT: INFO (hb): (hb.c:8642) node arrived bb9b7cf210a0102
Jun 27 2022 16:45:43 GMT: INFO (fabric): (fabric.c:2482) fabric: node bb9b7cf210a0102 arrived
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:160) NODE-ID bb95b91220a0102 CLUSTER-SIZE 7
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:238) cluster-clock: skew-ms 11
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:265) system-memory: free-kbytes 1558248 free-pct 1 heap-kbytes (72460413,88511368,116779008) heap-efficiency-pct 62.0
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:278) in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:299) fds: proto (994,132777746,132776752) heartbeat (7,1164,1157) fabric (168,4584,4416)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:307) heartbeat-received: self 2 foreign 1896953491
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:337) fabric-bytes-per-second: bulk (23583688,35759988) ctrl (6562,6211) meta (1400,1409) rw (1632918,1543647)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:360) early-fail: demarshal 0 tsvc-client 15201 tsvc-batch-sub 0 tsvc-udf-sub 0
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:376) batch-index: batches (1922750454,233072,12)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (1922983526 total) msec
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (00: 1921169686) (01: 0000701392) (02: 0000471429) (03: 0000293080)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (04: 0000147671) (05: 0000070526) (06: 0000045002) (07: 0000030297)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (08: 0000030188) (09: 0000011800) (10: 0000006774) (11: 0000005558)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:266) (12: 0000000120) (13: 0000000003)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:389) {test2} objects: all 33985392 master 13310011 prole 20675381 non-replica 0
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:444) {test2} migrations: remaining (263,221,526) active (1,0,0) complete-pct 2.81
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:465) {test2} memory-usage: total-bytes 62923237123 index-bytes 2175065088 sindex-bytes 2199407698 data-bytes 58548764337 used-pct 65.11
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:517) {test2} device-usage: used-bytes 60447949728 avail-pct 95
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:585) {test2} client: tsvc (0,3) proxy (555,0,32) read (39500824722,0,0,741801500) write (8614944386,19783,614) delete (12454423,0,1,17672660) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:635) {test2} batch-sub: tsvc (0,0) proxy (36,0,12) read (2992510013,0,0,3020503)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:662) {test2} scan: basic (90214,1620,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:686) {test2} query: basic (55282178096,12165) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:750) {test2} retransmits: migration 12208 client-read 0 client-write (0,60) client-delete (0,0) client-udf (0,0) batch-sub 0 udf-sub (0,0)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:785) {test2} special-errors: key-busy 63 record-too-big 17918
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-read (40242626222 total) msec
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (00: 40242597465) (01: 0000017936) (02: 0000007405) (03: 0000001806)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (04: 0000001185) (05: 0000000322) (06: 0000000063) (07: 0000000025)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:266) (08: 0000000002) (09: 0000000012) (10: 0000000001)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-write (8614964169 total) msec
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (00: 8308478071) (01: 0277638756) (02: 0020410826) (03: 0004604054)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (04: 0002235944) (05: 0001068045) (06: 0000287001) (07: 0000176658)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (08: 0000057847) (09: 0000004416) (10: 0000002535) (11: 0000000010)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:266) (12: 0000000006)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query (55282190261 total) msec
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (00: 55277803957) (01: 0002098036) (02: 0001068803) (03: 0000666989)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (04: 0000370901) (05: 0000141411) (06: 0000031785) (07: 0000005058)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (08: 0000001626) (09: 0000000292) (10: 0000000054) (11: 0000000025)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (12: 0000000013) (13: 0000000017) (14: 0000000653) (15: 0000000601)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (16: 0000000021) (17: 0000000003) (18: 0000000007) (19: 0000000001)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:266) (20: 0000000003) (21: 0000000005)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query-rec-count (10195924739 total) count
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (01: 7363953208) (02: 2057564886) (03: 0696637447) (04: 0063010442)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (05: 0008355227) (06: 0004445484) (07: 0001014606) (08: 0000535951)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (09: 0000303291) (10: 0000093732) (11: 0000007988) (12: 0000001101)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (13: 0000000028) (14: 0000000024) (15: 0000000011) (16: 0000000015)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:266) (17: 0000000013) (18: 0000001285)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:444) {test} migrations: remaining (176,159,352) active (0,0,0) complete-pct 32.73
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:585) {test} client: tsvc (0,0) proxy (0,0,0) read (495312379,0,0,2647816) write (98664489,81060,15) delete (250301,0,0,55989) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:662) {test} scan: basic (5639,23,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:45:52 GMT: INFO (info): (ticker.c:686) {test} query: basic (349361864,0) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:240) histogram dump: {test}-read (497960195 total) msec
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (00: 0497960013) (01: 0000000109) (02: 0000000038) (03: 0000000011)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (04: 0000000004) (05: 0000000009) (06: 0000000002) (07: 0000000004)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:266) (08: 0000000003) (09: 0000000002)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:240) histogram dump: {test}-write (98745549 total) msec
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (00: 0098260376) (01: 0000440271) (02: 0000014651) (03: 0000006381)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (04: 0000007308) (05: 0000007824) (06: 0000004358) (07: 0000003445)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (08: 0000000864) (09: 0000000053) (10: 0000000017) (11: 0000000001)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query (349361864 total) msec
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (00: 0349361358) (01: 0000000317) (02: 0000000073) (03: 0000000055)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:266) (04: 0000000050) (05: 0000000011)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query-rec-count (50590271 total) count
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:257) (01: 0049917514) (02: 0000324346) (03: 0000162011) (04: 0000172119)
Jun 27 2022 16:45:52 GMT: INFO (info): (hist.c:266) (05: 0000014281)
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:45:52 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:52 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:52 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:52 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:52 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:52 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:52 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:900) (repeated:200) Error while connecting: 111 (Connection refused)
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:900) (repeated:7) Error while connecting: 113 (No route to host)
Jun 27 2022 16:45:52 GMT: WARNING (socket): (socket.c:891) (repeated:73) Timeout while connecting
Jun 27 2022 16:45:52 GMT: INFO (drv_ssd): (drv_ssd.c:2185) {test2} /dev/vdb: used-bytes 60448233088 free-wblocks 2008439 write-q 0 write (548457506,18.5) defrag-q 0 defrag-read (548368576,2.7) defrag-write (265090565,1.3)
Jun 27 2022 16:45:56 GMT: INFO (hb): (hb.c:8066) node expired bb9b7cf210a0102
Jun 27 2022 16:45:56 GMT: INFO (fabric): (fabric.c:2486) fabric: node bb9b7cf210a0102 departed
Jun 27 2022 16:45:56 GMT: INFO (fabric): (fabric.c:923) fabric_node_disconnect(bb9b7cf210a0102)
Jun 27 2022 16:45:57 GMT: INFO (hb): (hb.c:8642) node arrived bb9b7cf210a0102
Jun 27 2022 16:45:57 GMT: INFO (fabric): (fabric.c:2482) fabric: node bb9b7cf210a0102 arrived
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:160) NODE-ID bb95b91220a0102 CLUSTER-SIZE 7
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:238) cluster-clock: skew-ms 11
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:265) system-memory: free-kbytes 1515008 free-pct 1 heap-kbytes (72727039,88612584,116801536) heap-efficiency-pct 62.3
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:278) in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:299) fds: proto (1072,132777840,132776768) heartbeat (7,1165,1158) fabric (168,4608,4440)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:307) heartbeat-received: self 2 foreign 1896953734
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:337) fabric-bytes-per-second: bulk (25695717,26696444) ctrl (213,226) meta (2,2) rw (3268059,3456974)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:360) early-fail: demarshal 0 tsvc-client 15201 tsvc-batch-sub 0 tsvc-udf-sub 0
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:376) batch-index: batches (1922750821,233072,12)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (1922983893 total) msec
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (00: 1921170050) (01: 0000701393) (02: 0000471431) (03: 0000293080)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (04: 0000147671) (05: 0000070526) (06: 0000045002) (07: 0000030297)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (08: 0000030188) (09: 0000011800) (10: 0000006774) (11: 0000005558)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:266) (12: 0000000120) (13: 0000000003)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:389) {test2} objects: all 34125984 master 13310128 prole 20815860 non-replica 0
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:444) {test2} migrations: remaining (256,215,512) active (1,1,0) complete-pct 5.42
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:465) {test2} memory-usage: total-bytes 63183673505 index-bytes 2184062976 sindex-bytes 2204445346 data-bytes 58795165183 used-pct 65.38
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:517) {test2} device-usage: used-bytes 60702210544 avail-pct 95
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:585) {test2} client: tsvc (0,3) proxy (555,0,32) read (39500829388,0,0,741801912) write (8614945735,19783,614) delete (12454423,0,1,17672660) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:635) {test2} batch-sub: tsvc (0,0) proxy (36,0,12) read (2992510767,0,0,3020503)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:662) {test2} scan: basic (90214,1620,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:686) {test2} query: basic (55282186649,12165) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:750) {test2} retransmits: migration 12208 client-read 0 client-write (0,60) client-delete (0,0) client-udf (0,0) batch-sub 0 udf-sub (0,0)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:785) {test2} special-errors: key-busy 63 record-too-big 17918
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-read (40242631300 total) msec
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (00: 40242602543) (01: 0000017936) (02: 0000007405) (03: 0000001806)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (04: 0000001185) (05: 0000000322) (06: 0000000063) (07: 0000000025)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:266) (08: 0000000002) (09: 0000000012) (10: 0000000001)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-write (8614965518 total) msec
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (00: 8308479375) (01: 0277638786) (02: 0020410833) (03: 0004604058)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (04: 0002235947) (05: 0001068045) (06: 0000287001) (07: 0000176659)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (08: 0000057847) (09: 0000004416) (10: 0000002535) (11: 0000000010)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:266) (12: 0000000006)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query (55282198814 total) msec
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (00: 55277812506) (01: 0002098040) (02: 0001068803) (03: 0000666989)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (04: 0000370901) (05: 0000141411) (06: 0000031785) (07: 0000005058)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (08: 0000001626) (09: 0000000292) (10: 0000000054) (11: 0000000025)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (12: 0000000013) (13: 0000000017) (14: 0000000653) (15: 0000000601)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (16: 0000000021) (17: 0000000003) (18: 0000000007) (19: 0000000001)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:266) (20: 0000000003) (21: 0000000005)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query-rec-count (10195926482 total) count
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (01: 7363954502) (02: 2057565169) (03: 0696637580) (04: 0063010461)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (05: 0008355239) (06: 0004445484) (07: 0001014608) (08: 0000535951)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (09: 0000303291) (10: 0000093732) (11: 0000007988) (12: 0000001101)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (13: 0000000028) (14: 0000000024) (15: 0000000011) (16: 0000000015)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:266) (17: 0000000013) (18: 0000001285)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:444) {test} migrations: remaining (176,159,352) active (0,0,0) complete-pct 32.73
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:585) {test} client: tsvc (0,0) proxy (0,0,0) read (495312379,0,0,2647816) write (98664489,81060,15) delete (250301,0,0,55989) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:662) {test} scan: basic (5639,23,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:46:02 GMT: INFO (info): (ticker.c:686) {test} query: basic (349361864,0) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:240) histogram dump: {test}-read (497960195 total) msec
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (00: 0497960013) (01: 0000000109) (02: 0000000038) (03: 0000000011)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (04: 0000000004) (05: 0000000009) (06: 0000000002) (07: 0000000004)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:266) (08: 0000000003) (09: 0000000002)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:240) histogram dump: {test}-write (98745549 total) msec
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (00: 0098260376) (01: 0000440271) (02: 0000014651) (03: 0000006381)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (04: 0000007308) (05: 0000007824) (06: 0000004358) (07: 0000003445)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (08: 0000000864) (09: 0000000053) (10: 0000000017) (11: 0000000001)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query (349361864 total) msec
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (00: 0349361358) (01: 0000000317) (02: 0000000073) (03: 0000000055)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:266) (04: 0000000050) (05: 0000000011)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query-rec-count (50590271 total) count
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:257) (01: 0049917514) (02: 0000324346) (03: 0000162011) (04: 0000172119)
Jun 27 2022 16:46:02 GMT: INFO (info): (hist.c:266) (05: 0000014281)
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:02 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:02 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:02 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:02 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:02 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:02 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:02 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:900) (repeated:200) Error while connecting: 111 (Connection refused)
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:900) (repeated:6) Error while connecting: 113 (No route to host)
Jun 27 2022 16:46:02 GMT: WARNING (socket): (socket.c:891) (repeated:74) Timeout while connecting
Jun 27 2022 16:46:10 GMT: INFO (clustering): (clustering.c:3510) clique based evicted nodes while updating preferred principal: bb9080b200a0102
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:160) NODE-ID bb95b91220a0102 CLUSTER-SIZE 7
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:238) cluster-clock: skew-ms 11
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:265) system-memory: free-kbytes 1482720 free-pct 1 heap-kbytes (72996676,88717276,116801536) heap-efficiency-pct 62.5
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:278) in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:299) fds: proto (1223,132778029,132776806) heartbeat (7,1166,1159) fabric (168,4608,4440)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:307) heartbeat-received: self 2 foreign 1896953974
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:337) fabric-bytes-per-second: bulk (25345357,27036156) ctrl (184,196) meta (0,0) rw (2643068,2563264)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:360) early-fail: demarshal 0 tsvc-client 15201 tsvc-batch-sub 0 tsvc-udf-sub 0
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:376) batch-index: batches (1922751250,233072,12)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (1922984322 total) msec
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (00: 1921170477) (01: 0000701394) (02: 0000471432) (03: 0000293080)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (04: 0000147671) (05: 0000070526) (06: 0000045002) (07: 0000030297)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (08: 0000030188) (09: 0000011800) (10: 0000006774) (11: 0000005558)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:266) (12: 0000000120) (13: 0000000003)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:389) {test2} objects: all 34264857 master 13310151 prole 20954707 non-replica 0
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:444) {test2} migrations: remaining (250,208,500) active (1,1,0) complete-pct 8.03
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:465) {test2} memory-usage: total-bytes 63439306974 index-bytes 2192950848 sindex-bytes 2209511876 data-bytes 59036844250 used-pct 65.65
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:517) {test2} device-usage: used-bytes 60951649392 avail-pct 95
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:585) {test2} client: tsvc (0,3) proxy (555,0,32) read (39500831663,0,0,741802051) write (8614946557,19783,614) delete (12454424,0,1,17672660) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:635) {test2} batch-sub: tsvc (0,0) proxy (36,0,12) read (2992511463,0,0,3020503)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:662) {test2} scan: basic (90214,1620,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:686) {test2} query: basic (55282192374,12165) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:750) {test2} retransmits: migration 12208 client-read 0 client-write (0,60) client-delete (0,0) client-udf (0,0) batch-sub 0 udf-sub (0,0)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:785) {test2} special-errors: key-busy 63 record-too-big 17918
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-read (40242633714 total) msec
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (00: 40242604957) (01: 0000017936) (02: 0000007405) (03: 0000001806)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (04: 0000001185) (05: 0000000322) (06: 0000000063) (07: 0000000025)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:266) (08: 0000000002) (09: 0000000012) (10: 0000000001)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-write (8614966340 total) msec
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (00: 8308480148) (01: 0277638820) (02: 0020410844) (03: 0004604061)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (04: 0002235947) (05: 0001068046) (06: 0000287001) (07: 0000176659)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (08: 0000057847) (09: 0000004416) (10: 0000002535) (11: 0000000010)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:266) (12: 0000000006)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query (55282204539 total) msec
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (00: 55277818228) (01: 0002098040) (02: 0001068805) (03: 0000666989)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (04: 0000370902) (05: 0000141411) (06: 0000031785) (07: 0000005058)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (08: 0000001626) (09: 0000000292) (10: 0000000054) (11: 0000000025)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (12: 0000000013) (13: 0000000017) (14: 0000000653) (15: 0000000601)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (16: 0000000021) (17: 0000000003) (18: 0000000007) (19: 0000000001)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:266) (20: 0000000003) (21: 0000000005)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query-rec-count (10195927460 total) count
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (01: 7363955156) (02: 2057565353) (03: 0696637697) (04: 0063010480)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (05: 0008355242) (06: 0004445485) (07: 0001014608) (08: 0000535951)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (09: 0000303291) (10: 0000093732) (11: 0000007988) (12: 0000001101)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (13: 0000000028) (14: 0000000024) (15: 0000000011) (16: 0000000015)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:266) (17: 0000000013) (18: 0000001285)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:444) {test} migrations: remaining (176,159,352) active (0,0,0) complete-pct 32.73
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:585) {test} client: tsvc (0,0) proxy (0,0,0) read (495312379,0,0,2647816) write (98664489,81060,15) delete (250301,0,0,55989) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:662) {test} scan: basic (5639,23,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:46:12 GMT: INFO (info): (ticker.c:686) {test} query: basic (349361864,0) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:240) histogram dump: {test}-read (497960195 total) msec
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (00: 0497960013) (01: 0000000109) (02: 0000000038) (03: 0000000011)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (04: 0000000004) (05: 0000000009) (06: 0000000002) (07: 0000000004)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:266) (08: 0000000003) (09: 0000000002)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:240) histogram dump: {test}-write (98745549 total) msec
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (00: 0098260376) (01: 0000440271) (02: 0000014651) (03: 0000006381)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (04: 0000007308) (05: 0000007824) (06: 0000004358) (07: 0000003445)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (08: 0000000864) (09: 0000000053) (10: 0000000017) (11: 0000000001)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query (349361864 total) msec
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (00: 0349361358) (01: 0000000317) (02: 0000000073) (03: 0000000055)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:266) (04: 0000000050) (05: 0000000011)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query-rec-count (50590271 total) count
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:257) (01: 0049917514) (02: 0000324346) (03: 0000162011) (04: 0000172119)
Jun 27 2022 16:46:12 GMT: INFO (info): (hist.c:266) (05: 0000014281)
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:959) (repeated:40) Error while connecting socket to masked:3002
Jun 27 2022 16:46:12 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:12 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:12 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:12 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:12 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:12 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:12 GMT: WARNING (hb): (hb.c:4882) (repeated:40) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:900) (repeated:200) Error while connecting: 111 (Connection refused)
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:900) (repeated:6) Error while connecting: 113 (No route to host)
Jun 27 2022 16:46:12 GMT: WARNING (socket): (socket.c:891) (repeated:74) Timeout while connecting
Jun 27 2022 16:46:12 GMT: INFO (drv_ssd): (drv_ssd.c:2185) {test2} /dev/vdb: used-bytes 60958933328 free-wblocks 2007948 write-q 0 write (548458161,32.8) defrag-q 0 defrag-read (548368742,8.3) defrag-write (265090646,4.1)
Jun 27 2022 16:46:14 GMT: INFO (hb): (hb.c:8066) node expired bb9b7cf210a0102
Jun 27 2022 16:46:14 GMT: INFO (fabric): (fabric.c:2486) fabric: node bb9b7cf210a0102 departed
Jun 27 2022 16:46:14 GMT: INFO (fabric): (fabric.c:923) fabric_node_disconnect(bb9b7cf210a0102)
Jun 27 2022 16:46:17 GMT: INFO (hb): (hb.c:8642) node arrived bb9b7cf210a0102
Jun 27 2022 16:46:17 GMT: INFO (fabric): (fabric.c:2482) fabric: node bb9b7cf210a0102 arrived
Jun 27 2022 16:46:19 GMT: INFO (clustering): (clustering.c:5646) moved self node to orphan state
Jun 27 2022 16:46:19 GMT: INFO (clustering): (clustering.c:5989) sent cluster join request to bb98fc7220a0102
Jun 27 2022 16:46:21 GMT: INFO (clustering): (clustering.c:5989) sent cluster join request to bb9b7cf210a0102
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:160) NODE-ID bb95b91220a0102 CLUSTER-SIZE 7
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:238) cluster-clock: skew-ms 11
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:265) system-memory: free-kbytes 1079748 free-pct 1 heap-kbytes (73598226,89185956,117268480) heap-efficiency-pct 62.8
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:278) in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:299) fds: proto (1324,132778192,132776868) heartbeat (7,1168,1161) fabric (168,4632,4464)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:307) heartbeat-received: self 2 foreign 1896954220
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:337) fabric-bytes-per-second: bulk (18458260,23181687) ctrl (7612,6945) meta (2,2) rw (6029280,5467211)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:360) early-fail: demarshal 0 tsvc-client 15201 tsvc-batch-sub 0 tsvc-udf-sub 0
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:376) batch-index: batches (1922751843,233072,12)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:240) histogram dump: batch-index (1922984915 total) msec
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (00: 1921171047) (01: 0000701396) (02: 0000471433) (03: 0000293087)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (04: 0000147679) (05: 0000070528) (06: 0000045004) (07: 0000030297)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (08: 0000030189) (09: 0000011800) (10: 0000006774) (11: 0000005558)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:266) (12: 0000000120) (13: 0000000003)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:389) {test2} objects: all 34386265 master 13310337 prole 21075928 non-replica 0
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:444) {test2} migrations: remaining (246,204,492) active (0,0,0) complete-pct 9.64
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:465) {test2} memory-usage: total-bytes 63669232448 index-bytes 2200720960 sindex-bytes 2213696484 data-bytes 59254815004 used-pct 65.89
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:517) {test2} device-usage: used-bytes 61176408480 avail-pct 95
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:585) {test2} client: tsvc (0,3) proxy (555,0,32) read (39500840040,0,0,741802370) write (8614948785,19783,614) delete (12454426,0,1,17672660) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:635) {test2} batch-sub: tsvc (0,0) proxy (36,0,12) read (2992512736,0,0,3020503)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:662) {test2} scan: basic (90214,1620,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:686) {test2} query: basic (55282207925,12165) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:750) {test2} retransmits: migration 12208 client-read 0 client-write (0,60) client-delete (0,0) client-udf (0,0) batch-sub 0 udf-sub (0,0)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:785) {test2} special-errors: key-busy 63 record-too-big 17918
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-read (40242642410 total) msec
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (00: 40242613624) (01: 0000017939) (02: 0000007405) (03: 0000001806)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (04: 0000001187) (05: 0000000332) (06: 0000000067) (07: 0000000028)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:266) (08: 0000000009) (09: 0000000012) (10: 0000000001)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-write (8614968568 total) msec
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (00: 8308482271) (01: 0277638879) (02: 0020410861) (03: 0004604068)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (04: 0002235947) (05: 0001068046) (06: 0000287006) (07: 0000176659)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (08: 0000057860) (09: 0000004420) (10: 0000002535) (11: 0000000010)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:266) (12: 0000000006)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query (55282220090 total) msec
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (00: 55277833743) (01: 0002098044) (02: 0001068806) (03: 0000666992)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (04: 0000370909) (05: 0000141412) (06: 0000031785) (07: 0000005058)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (08: 0000001637) (09: 0000000301) (10: 0000000054) (11: 0000000025)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (12: 0000000013) (13: 0000000017) (14: 0000000653) (15: 0000000601)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (16: 0000000021) (17: 0000000003) (18: 0000000007) (19: 0000000001)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:266) (20: 0000000003) (21: 0000000005)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:240) histogram dump: {test2}-query-rec-count (10195929923 total) count
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (01: 7363956957) (02: 2057565746) (03: 0696637927) (04: 0063010514)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (05: 0008355244) (06: 0004445485) (07: 0001014611) (08: 0000535951)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (09: 0000303291) (10: 0000093732) (11: 0000007988) (12: 0000001101)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (13: 0000000028) (14: 0000000024) (15: 0000000011) (16: 0000000015)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:266) (17: 0000000013) (18: 0000001285)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:389) {test} objects: all 0 master 0 prole 0 non-replica 0
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:444) {test} migrations: remaining (176,159,352) active (0,0,0) complete-pct 32.73
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:465) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:585) {test} client: tsvc (0,0) proxy (0,0,0) read (495312379,0,0,2647816) write (98664489,81060,15) delete (250301,0,0,55989) udf (0,0,0) lang (0,0,0,0)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:662) {test} scan: basic (5639,23,0) aggr (0,0,0) udf-bg (0,0,0)
Jun 27 2022 16:46:22 GMT: INFO (info): (ticker.c:686) {test} query: basic (349361864,0) aggr (0,0) udf-bg (0,0)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:240) histogram dump: {test}-read (497960195 total) msec
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (00: 0497960013) (01: 0000000109) (02: 0000000038) (03: 0000000011)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (04: 0000000004) (05: 0000000009) (06: 0000000002) (07: 0000000004)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:266) (08: 0000000003) (09: 0000000002)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:240) histogram dump: {test}-write (98745549 total) msec
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (00: 0098260376) (01: 0000440271) (02: 0000014651) (03: 0000006381)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (04: 0000007308) (05: 0000007824) (06: 0000004358) (07: 0000003445)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (08: 0000000864) (09: 0000000053) (10: 0000000017) (11: 0000000001)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query (349361864 total) msec
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (00: 0349361358) (01: 0000000317) (02: 0000000073) (03: 0000000055)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:266) (04: 0000000050) (05: 0000000011)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:240) histogram dump: {test}-query-rec-count (50590271 total) count
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:257) (01: 0049917514) (02: 0000324346) (03: 0000162011) (04: 0000172119)
Jun 27 2022 16:46:22 GMT: INFO (info): (hist.c:266) (05: 0000014281)
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:959) (repeated:37) Error while connecting socket to masked:3002
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:959) (repeated:36) Error while connecting socket to masked:3002
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:959) (repeated:36) Error while connecting socket to masked:3002
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:959) (repeated:37) Error while connecting socket to masked:3002
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:959) (repeated:37) Error while connecting socket to masked:3002
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:959) (repeated:37) Error while connecting socket to masked:3002
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:959) (repeated:37) Error while connecting socket to masked:3002
Jun 27 2022 16:46:22 GMT: WARNING (hb): (hb.c:4882) (repeated:37) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:22 GMT: WARNING (hb): (hb.c:4882) (repeated:36) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:22 GMT: WARNING (hb): (hb.c:4882) (repeated:36) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:22 GMT: WARNING (hb): (hb.c:4882) (repeated:37) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:22 GMT: WARNING (hb): (hb.c:4882) (repeated:37) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:22 GMT: WARNING (hb): (hb.c:4882) (repeated:37) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:22 GMT: WARNING (hb): (hb.c:4882) (repeated:37) could not create heartbeat connection to node {masked:3002}
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:900) (repeated:184) Error while connecting: 111 (Connection refused)
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:900) (repeated:4) Error while connecting: 113 (No route to host)
Jun 27 2022 16:46:22 GMT: WARNING (socket): (socket.c:891) (repeated:69) Timeout while connecting
````we had THP turned off if this helps in debugging.
cat /proc/meminfo
MemTotal: 99009560 kB
MemFree: 13786636 kB
MemAvailable: 18895604 kB
Buffers: 149912 kB
Cached: 6740264 kB
SwapCached: 0 kB
Active: 80052808 kB
Inactive: 4634552 kB
Active(anon): 78413012 kB
Inactive(anon): 378020 kB
Active(file): 1639796 kB
Inactive(file): 4256532 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 304 kB
Writeback: 0 kB
AnonPages: 77797316 kB
Mapped: 128076 kB
Shmem: 993848 kB
Slab: 103136 kB
SReclaimable: 78824 kB
SUnreclaim: 24312 kB
KernelStack: 9296 kB
PageTables: 179300 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49504780 kB
Committed_AS: 3953168 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 200540 kB
DirectMap2M: 6090752 kB
DirectMap1G: 96468992 kB
but min_free_kbytes are too low.
cat /proc/sys/vm/min_free_kbytes
67584
cat /proc/sys/vm/swappiness
60
looking at best practice doc, i do realise that these are not the best possible values. but imo it would help understanding what might be happening. Also we do use secondary indexes.
Here are the two lines in the log file to scrutinize for checking on memory:
So, there is indeed pretty much no memory left (free-pct 1) and we have allocated ~69 GiB of heap (72063950 kbytes) but there is some fragmentation (heap-efficiency-pct 61.7). We have allocated over 84GiB of heap and even more has been mapped.
You may have already seen or been pointed at this article: Understanding linux memory usage reporting.
Now to what is allocating the memory, the second line breaks it down… for the Primary Index (PI), if using the Enterprise Edition, that will be in shared memory and not counted in the heap numbers, but if Community Edition, it would be on the heap… doesn’t really matter since we have barely 2GiB for the PI. Similar for the secondary index, but for the data, we have over 50GiB. It therefore seems we may have some fragmentation of the heap for the data itself… restarting the node should recover the memory. If data is, for example, slowly growing over time, it can cause fragmentation (refer to this article: FAQ Does migration cause memory fragmentation when using Data In Memory?).
You may want (if not done) to upgrade (there were fairly recent changes that updated the JEMalloc library used) and this is also something that may be improved in the future to try to avoid such fragmentation.
Hope this helps a bit.