Community Edition - Service can't recover after oom when reach the hwm

Operations
1

we used memory mode , with data store in disk ; after oom , we restart the service , and the as load data from disk into memory but when reach the high-water-mark the recover progress become very slow , and the log say

Sep 11 2019 12:38:19 GMT: INFO (nsup): (nsup.c:1109) {dmp} cold start building evict histogram ...
Sep 11 2019 12:38:22 GMT: INFO (drv_ssd): (drv_ssd.c:3814) {dmp} loaded 330968053 objects, /pdata1/aerospike.dat 26%
Sep 11 2019 12:38:27 GMT: INFO (drv_ssd): (drv_ssd.c:3814) {dmp} loaded 330968053 objects, /pdata1/aerospike.dat 26%
Sep 11 2019 12:38:28 GMT: INFO (nsup): (nsup.c:1160) {dmp} cold start found 224794 records eligible for eviction at evict-ttl 39835
Sep 11 2019 12:38:32 GMT: INFO (drv_ssd): (drv_ssd.c:3814) {dmp} loaded 330870300 objects, /pdata1/aerospike.dat 26%
Sep 11 2019 12:38:36 GMT: INFO (nsup): (nsup.c:1178) {dmp} cold start evicted 224794 records, found 283196833 0-void-time records
Sep 11 2019 12:38:37 GMT: INFO (drv_ssd): (drv_ssd.c:3814) {dmp} loaded 330762149 objects, /pdata1/aerospike.dat 26%
Sep 11 2019 12:38:37 GMT: WARNING (nsup): (nsup.c:684) {dmp} breached eviction hwm (memory), memory sz:51539662152 (21179634112 + 0 + 30360028040) hwm:51539607552, index-device sz:0 hwm:0, disk sz:45455825216 hwm:171798691840
2

While loading records from disk, you have breached your memory hwm which requires Aerospike to evict records. This typically means that NSUP was falling behind with eviction rounds. This will cause cold-start to take longer.