Aerospike 5.4 to 7.2 Namespace tuning

Operations Tuning
1

Hi! We are currently running aerospike 5.4, with the intention of migrating to 7.2.

Details below, but the main question is this: Is there a suggested 7.2+ configuration for systems that are memory constrained? Or is there a way to match the old behavior where the stop writes % was based on the index memory used, instead of free system memory?

The changes in how namespace memory usage is handled in newer versions are causing complications for us since the suggested values in our environment lead to stop-writes conditions even in a initial state.

This is mostly due to resource constraints in our use case, since we have limited resources to work with and all of the services for our app need to share the same instances. We have 8G instances, and while memory does get tight, on 5.4 we have never encountered a stop-write condition due to memory usage.

Here is our current 5.4 namespace config:

namespace example {
    replication-factor 2
    memory-size 400M
    storage-engine device {
        file /var/tb/aerospike/example
        filesize 100G
        data-in-memory false
    }
}

Following the best practices, particularly after setting min_free_kbytes, the memory availability starts off in the 10% range, leading to almost immediate stop-writes. Lowering min_free_kbytes frees this up, but we would still need to operate under the expectation that writes may fail if memory usage increases.

  1. Would it be possible to use indexes-memory-budget as a replacement for memory-size, and disable stop-writes-sys-memory-pct by setting it to 0? Is there a way to stop writes when the memory budget is reached, instead of total system memory? The docs for indexes-memory-budget indicate that stop-writes will occur if it is breached, but the wording in unclear if it will behave that way without evict-indexes-memory-pct.
namespace example {
    replication-factor 2
    storage-engine device {
        file /var/tb/aerospike/example
        filesize 100G
    }
    index-stage-size 128M
    stop-writes-sys-memory-pct 0
    indexes-memory-budget 400M
}
  1. Currently we expect that memory may be consumed up to the point of swapping, expecting degraded performance but no write failures. Will memory consumption up to the point of swapping cause any issues in 7.0+ with stop-writes-sys-memory-pct set to 0? The docs for that value advise against setting too low to avoid OOM conditions, but if there is swap available that shouldn’t happen.

Thanks for reading

2

Yes, this seems equivalent. The old memory-size config also covered data-in-memory but it also tended to be confusing to user who expected it to be a hard limit. When storage-engine memory was overhauled to become warm-restartable in EE, it no longer made sense for memory-size to include the data. Since the definition needed to be changed we also took the opportunity to rename it to indexes-memory-budget where budget was meant to convey the soft limit nature of the config.

3

Thanks, we’ll give that a try.

As mentioned, min_free_kbytes brings down usable memory as well. Can we leave that at the default since we are using a device backed namespace?