We ran into HWM breach and cluster went down. I don’t see asd process anymore on servers. Ideally it should either keep evicting or pause writes when it reaches 90%. But why is the node not responding to hearbeat and cluster size getting dropped to 0?
Build - C-3.15.1.3
Apr 08 2018 11:41:06 GMT: INFO (info): (ticker.c:435) {nano} memory-usage: total-bytes 23729610339 index-bytes 435985216 sindex-bytes 0 data-bytes 23293625123 used-pct 85.00
Apr 08 2018 11:41:06 GMT: INFO (info): (ticker.c:465) {nano} device-usage: used-bytes 24223068928 avail-pct 65
Apr 08 2018 11:41:06 GMT: INFO (info): (ticker.c:534) {nano} client: tsvc (0,1) proxy (186,0,0) read (3798065,0,0,9966748) write (3879032,847,0) delete (0,0,0,0) udf (0,0,0) lang (0,0,0,0)
Apr 08 2018 11:41:06 GMT: INFO (info): (ticker.c:684) {nano} retransmits: migration 0 client-read 0 client-write (0,3) client-delete (0,0) client-udf (0,0) batch-sub 0 udf-sub (0,0) nsup 0
Apr 08 2018 11:41:06 GMT: INFO (info): (hist.c:139) histogram dump: {nano}-read (13764813 total) msec
Apr 08 2018 11:41:06 GMT: INFO (info): (hist.c:156) (00: 0013738942) (01: 0000023435) (02: 0000001996) (03: 0000000425)
Apr 08 2018 11:41:06 GMT: INFO (info): (hist.c:165) (04: 0000000011) (05: 0000000002) (07: 0000000002)
Apr 08 2018 11:41:06 GMT: INFO (info): (hist.c:139) histogram dump: {nano}-write (3879879 total) msec
Apr 08 2018 11:41:06 GMT: INFO (info): (hist.c:156) (00: 0003608154) (01: 0000204420) (02: 0000039144) (03: 0000012968)
Apr 08 2018 11:41:06 GMT: INFO (info): (hist.c:156) (04: 0000008618) (05: 0000005165) (06: 0000001168) (07: 0000000152)
Apr 08 2018 11:41:06 GMT: INFO (info): (hist.c:165) (08: 0000000082) (09: 0000000008)
Apr 08 2018 11:41:13 GMT: WARNING (socket): (socket.c:749) Error while connecting: 111 (Connection refused)
Apr 08 2018 11:41:13 GMT: WARNING (socket): (socket.c:808) Error while connecting socket to 11.242.116.114:3002
Apr 08 2018 11:41:13 GMT: WARNING (hb): (hb.c:4669) could not create heartbeat connection to node {11.242.116.114:3002}
Apr 08 2018 11:41:14 GMT: INFO (drv_ssd): (drv_ssd.c:2072) {nano} /var/lib/aerospike/nano.dat: used-bytes 24223664640 free-wblocks 586236 write-q 0 write (206175,1.1) defrag-q 0 defrag-read (117607,0.1) defrag-write (39713,0.1)