Here we look at some of the default values used by Aerospike and explain them further.
high-water-memory-pct – this is set at 60% by default. Is there a reason why this is so low and are we not wasting disk space? Would not 85% or 90% be better?
The recommended setting for high-water-memory-pct is 60%. This is a safe setting for clusters with less than 6 nodes. If a node is down for maintenance or other issues, the cluster will redistribute the data. During this redistribution phase memory usage on a particular node can fluctuate quickly, especially on smaller clusters. In a 3 node cluster if a single node drops the amount of data on the remaining two will increase by 50% after migrations. During migrations the two nodes may in worst case increase to double their original size. A 60% high water mark is meant to protect the system from running out of memory in the near worst case scenario of a 3 node cluster.
For larger clusters you can push this setting higher, however be cautions of going above 80% though. An added benefit of our recommendation is that it provides protection against memory fragmentation. Pushing past 80% doesn’t allow much space for fragmentation. In Aerospike 3 we have changed memory allocation algorithms and introduced strategies to reduce the rate of memory fragmentation.
Lastly if a node has a hardware failure and a replacement cannot be provisioned for an extended duration, you can temporarily increase the high-water-mark until the node can be replaced.
high-water-disk-pct – should this always be the same as the memory-pct or do we need to keep some free space on disk to allow for defrag.
The recommended setting for high-water-disk-pct is 50% which means it is not necessary for this to be the same as high-water-memory-pct. Like memory we recommend 50% due to disk fragmentation and sudden write load spikes potentially caused by a downed node.
This default value is loosely coupled to the
defrag-lwm-pct setting and has to do with write amplification, which can impact performance. For further details, refer to the following article: FAQ - Why is high-water-disk-pct set to 50%?.
Should memory-size and file-size be the same if using a file to persist? What should the relationship between these values be?
As a rule of thumb (since 2.7.0 and 3.1.3), file-size should be a factor of 4 larger than memory-size. The additional overhead is due to the size of records rounding up to the nearest record block (rblock) size as well as the additional space required for defrag. Lastly the larger your records the closer the in memory size will be to the data on disk which would reduce the amount of disk storage required. If you average record falls in the range of 512 1024b then disk storage would be 12-24% larger than memory storage size. In which case the file-size should be a minimum of 2.12 to 2.24 times larger than your memory-size.
File persistence is not recommended. What is the expected downside with using a file (we are not planning on doing so for the final production system, but would like to understand the downside here).
As long as data-in-memory is set, the downside would be slower writes. If data-in-memory is not set, reads will also be slower. You can decrease this effect some by mounting the disk with noatime.