I have a query regarding evictions in aerospike. For one of my persistent namespace(with data-in-memory false and high-water-memory-pct 60%), i found eviction of objects were happening for 2 of the nodes due to a breach in high-water-memory-pct(from amc we found this).
Currently i am of the impression that since indices are stored in memory, a breach of HWM on memory is actually evicting objects from the disk. Is my understanding correct?
Records will be evicted, regardless of where they are stored - disk or memory, if - 1) they have finite TTL (live-forever records are never evicted OR if they belong to a set where you have disabled eviction for that set by configuration), 2) either storage device high water mark (HWM) default 50% used is breached or memory usage HWM - default 60% - is breached. For memory HWM breach, what counts is the % of the declared memory size in the namespace - for e.g. if you set memory-size to 4G, once memory usage exceeds 2.4G, you have breached the memory HWM. What counts towards memory usage for HWM calculation? - 1) memory used by primary index, 2) memory used by data (if data or copy is being stored in memory) and 3) memory for any secondary indexes. All other memory usages such as for max-write-cache, post-write-queue, sprigs etc … are part of overall memory overhead and should definitely considered for sizing your instance but don’t contribute towards HWM breach. The configured memory towards a namespace and its HWM setting can be adjusted dynamically on a running cluster for temporary relief but if you are breaching your HWM - you need to revisit your capacity planning and sizing.