How To Tune the linux OS for Aerospike servers - best practices

How To Tune the linux OS for Aerospike servers - best practices

Context

When running Aerospike, overall performance and responsiveness can be improved by tuning some of the kernel parameters. This article describes what can be tuned, and suggests some optimum values.

Method

The following parameters can be used to tune the operating system for Aerospike usage.

min_free_kbytes

The min_free_kbytes parameter specified determines how much RAM should always be kept free, away from system r/w page caches. Setting this parameter will improve the overall system stability and allow Aerospike to work more efficiently.

The default for min_free_kbytes, while suitable for most applications, is too low for a low latency database such as Aerospike. In order to take full advantage of system caching and at the same time allow Aerospike to perform memory allocations without delay, it is adviseable to set this paramter to just over 1GB. This gives at least 1GB for Aerospike key allocation in shared memory and at least another 256MB RAM for all other purposes including the OS (i.e. a value of 1310720).

Specific details on how to set the parameter are documented in the Tuning Kernel Memory for Performance article.

swappiness

Aerospike advises either reducing swappiness to 0 or not using swap altogether. For low-latency operations, using swap to any extent will drastically slow down performance. Reducing swappiness will also improve system behaviour in terms of swapping out dirty and clean page caches.

Another parameter aimed at relaiming memory exists, zone_reclaim. Aerospike advises that this is disabled as it causes aggressive reclaims and memory scans. The default for zone_reclaim is disabled for most modern Linux distributions.

Details on how to set swappiness to 0 and ensure that zone_reclaim is disabled are documented in the Tuning Kernel Memory for Performance article.

AWS only - ENA (enhanced networking adapter)

AWS has a network driver called ENA. Installing and using this driver drastically improves the networking speed of AWS instances.

ENA is enabled if the module concerned is in use, this is checked as follows:

$ ethtool -i eth0 | grep ixgbevf

If the module is not being used, it can be installed or upgraded, by following this manual.

THP - transparent huge pages

In order to improve overall system responsiveness and allocation speed, The Linux kernel has a feature called THP, Transparent Huge Pages Unfortunately, for high-throughput and low-latency databases, which perform multiple small allocations, this can be counter productive. Having THP can cause the system to run out of RAM, with similar symptoms to a memory leak. Another issue is latency caused by THP defragmentation page locking.

In order to disable THP, refer to the Disabling Transparent Huge Pages for Aerospike article.

IRQ Balancing

Some network drivers will automatically balance network cards. Unfortunately, if they do not, multiple queues for a network interface could reside on a single CPU core. If this happens the irq or soft sides of a single core can use up to 100% of that core. This will be visible in the output of the mpstat command. If that happens, IRQ balancing should be enabled.

On newer kernels, installing the irq balance package should be enough for the correct balancing to happen. Unfortunately, on older kernels, this may not work as expected.

In order to check and reassign IRQ balance manually for network card queues, please follow this manual

Please note that if there is only one queue for a network interface in the /proc/interrupts file, the network card driver must be checked as more queues should be added. The network card vendor can advise on this.

The following article discusses how to read /proc/interrupts and what the values inside it mean.

Notes

The above steps make use of changing system-wide kernel parameters. While performing these should be safe, it is advisable to make said changes in a development environment first, to ensure no unforeseen effects for the particular use case and workloads.

Keywords

ENA THP IRQ BALANCING MIN_FREE_KBYTES SWAPPINESS ENHANCED NETWORKING TRANSPARENT HUGE PAGES

Timestamp

June 24 2019

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.