My cluster is 3 nodes of 16CPU / 32G RAM
Namespace config:
$ asinfo -v "get-config:context=namespace;id=redis"
repl-factor=2;memory-size=18439208960;default-ttl=3600;enable-xdr=false;sets-enable-xdr=true;ns-forward-xdr-writes=false;allow-nonxdr-writes=true;allow-xdr-writes=true;{redis}-read-hist-track-back=300;{redis}-read-hist-track-slice=10;{redis}-read-hist-track-thresholds=1,8,64;{redis}-query-hist-track-back=300;{redis}-query-hist-track-slice=10;{redis}-query-hist-track-thresholds=1,8,64;{redis}-udf-hist-track-back=300;{redis}-udf-hist-track-slice=10;{redis}-udf-hist-track-thresholds=1,8,64;{redis}-write-hist-track-back=300;{redis}-write-hist-track-slice=10;{redis}-write-hist-track-thresholds=1,8,64;cold-start-evict-ttl=4294967295;conflict-resolution-policy=generation;data-in-index=false;disallow-null-setname=false;enable-benchmarks-batch-sub=false;enable-benchmarks-read=false;enable-benchmarks-udf=false;enable-benchmarks-udf-sub=false;enable-benchmarks-write=false;enable-hist-proxy=false;evict-hist-buckets=10000;evict-tenths-pct=5;high-water-disk-pct=50;high-water-memory-pct=85;ldt-enabled=false;ldt-gc-rate=0;ldt-page-size=8192;max-ttl=2678400;migrate-order=5;migrate-retransmit-ms=5000;migrate-sleep=1;obj-size-hist-max=100;partition-tree-locks=8;partition-tree-sprigs=64;read-consistency-level-override=one;single-bin=false;stop-writes-pct=98;tomb-raider-eligible-age=86400;tomb-raider-period=86400;write-commit-level-override=master;storage-engine=memory;sindex.num-partitions=32;geo2dsphere-within.strict=true;geo2dsphere-within.min-level=1;geo2dsphere-within.max-level=30;geo2dsphere-within.max-cells=12;geo2dsphere-within.level-mod=1;geo2dsphere-within.earth-radius-meters=6371000
Number of record on the node (master / replicas) : 29,047,684 28,714,358
When I change the hwm to start eviction: asinfo -v "set-config:context=namespace;id=redis;high-water-memory-pct=70"
I have following error:
$ tail -f /var/log/aerospike/aerospike.log | grep thr_nsup
May 31 2017 01:37:56 GMT: INFO (nsup): (thr_nsup.c:1109) {redis} Records: 29136407, 0 0-vt, 611129(542999255) expired, 0(0) evicted, 0(0) set deletes. Evict ttl: 0. Waits: 0,0,85089. Total time: 119675 ms
May 31 2017 01:37:57 GMT: INFO (nsup): (thr_nsup.c:1400) nsup clear waits: 1617
May 31 2017 01:37:58 GMT: INFO (nsup): (thr_nsup.c:1181) {redis} nsup start
May 31 2017 01:38:13 GMT: WARNING (nsup): (thr_nsup.c:1057) {redis} no records below eviction void-time 233890678 - threshold bucket 0, width 9 sec, count 717063 > target 145709 (0.5 pct)
May 31 2017 01:40:00 GMT: INFO (nsup): (thr_nsup.c:1109) {redis} Records: 29141821, 0 0-vt, 620427(543619682) expired, 0(0) evicted, 0(0) set deletes. Evict ttl: 0. Waits: 0,0,86766. Total time: 121241 ms
May 31 2017 01:40:01 GMT: INFO (nsup): (thr_nsup.c:1400) nsup clear waits: 1635
May 31 2017 01:40:02 GMT: INFO (nsup): (thr_nsup.c:1181) {redis} nsup start
May 31 2017 01:40:17 GMT: WARNING (nsup): (thr_nsup.c:1057) {redis} no records below eviction void-time 233890802 - threshold bucket 0, width 9 sec, count 720330 > target 145685 (0.5 pct)
Eviction histogram
$ asinfo -v "hist-dump:ns=redis;hist=ttl" redis:ttl=100,864,2600428,1265011,1353508,1621274,710872,356439,365494,376524,311931,231012,235641,226260,229141,227373,228915,231406,245566,241468,243034,246931,254193,262570,275493,303319,301944,290748,289144,299810,302100,300799,310149,301445,280763,248216,250259,155079,173719,233837,250389,259380,261085,262715,243071,256577,245353,234971,244574,249289,256382,253645,247494,247154,248927,247022,243189,251188,244630,241392,236438,233808,232191,254283,274978,281376,293655,318017,368619,397570,417689,452757,492711,549978,603988,701842,970276,64459,31436,28307,25963,25380,24727,25432,25734,25479,24617,24121,24712,24799,22767,25176,24396,24414,23210,26266,28708,29699,29449,28103,26132,27917;
Changing params: asinfo -v "set-config:context=namespace;id=redis;evict-tenths-pct=100"
, I’m able to have some eviction.
The change from 0.5 to 100 seems huge. What do you think?