The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
Best practices for deploying Aerospike on VMware
Abstract
Deployment of high-performance low-latency distributed databases, such as Aerospike, on VMware may require some tuning, without which, the database may be prone to latency and other stability issues.
Memory and vSwap
It is best to turn off VMware vSwap for Aerospike VMs. Leaving this option switched on may cause the VM to exhibit latency as the host machine attempts to swap memory in and out. With respect to memory, the best approach involves performing a 100% Memory Reservation, by entering the amount of RAM required in the VM configuration and selecting Reserve all guest memory (All locked). This has the added benefit of avoiding physical memory fragmentation, further removing the potential for latency on the VM.
Refer to this document for more information.
vMotion
VMware vMotion allows the VMs to be automatically moved and rebalanced across an ESXi cluster. For a distributed database, such moves may cause VM stun, which is where the VM momentarily freezes, resulting in the database nodes leaving the database cluster during the period of the stun. This can also result in client application timeouts and errors during the period of the stun.
While it is much more efficient, from an Aerospike perspective, to run with vMotion disabled for the Aerospike VMs, if this is not possible due to policy reasons, the following should be taken into account:
Spread VMs across Fault Domains. vSphere has a concept of VM ↔ VM Anti-Affinity policy that spreads a set of VMs (a typical use case being sibling replicas in a distributed system) across ESXi hosts in a vSphere cluster. Since Aerospike is distributed and retains the required data replication factor, to ensure data will not be lost if a host machine dies, configure Aerospike VM affinity so that vMotion will not put any two Aerospike VMs on the same physical host. This is covered in greater detail here.
Upgrade to vSphere 7. If running on older versions, vMotion may stun high-performing VMs, such as an Aerospike database, for longer periods of time to migrate the virtual machine. This has been greatly improved in version 7. If possible, upgrade to take advantage of the new features and lower stun time. Refer to this document for details.
Ensure RARP packets are sent when VMs are migrated. As the virtual machines migrate, their routing also changes. All switches must be notified to remove/update the ARP entry (caching of MAC<->IP tables) so that they may rediscover the new route to the migrated host. This enables vMotion to work correctly, without long network downtime. It is important that all switches support ARP notifications in a VMware-compatible manner and that the vMotion’s notify switches option is enabled to send such packets. See here for further reading.
Avoid saturation of links by vMotion traffic. While vMotion will, by default, only saturate 10Gbps, it is possible to configure it to saturate larger links. As vMotion needs to transfer many GB of data for the VM RAM to be migrated, if possible, configure a dedicated link for vMotion fabric use. Such a dedicated link should use preferably a large MTU to assist these transfers. Jumbo packets with an MTU of 9000 are highly recommended.
VMware tools
Always use the latest compatible VMware tools on the virtual machine. This ensures that the virtual network adapter driver is running the latest available version with stability and speed improvements.
Noisy neighbours
Optionally, turn off paravirtualization for Aerospike VM if the host is affecting the VM due to other VMs putting load on the physical machine. Under certain heavy IO load conditions from multiple VMs, this option causes the VM to be 20% slower but improves stability and reliability of performance (the performance will be more linear). If paravirtualization is off, do not use vMotion, as vMotion will be extremely slow. See this VMWare KB article for more information.
Snapshotting
Virtual machine snapshots will affect the running VMs. This is due to the inherent nature of such snapshots. Copy-on-write snapshots will further affect VM performance once the snapshot is complete, as two drive operations will happen for each write (data is copied on fresh write). For performance reasons, it is adviseable to disable snapshotting of Aerospike VMs.
If using snapshots, ensure that the VM disks are snapshot only, not the RAM (memory). Enabling memory snapshot (to recover point-in-time running VM) will cause the machine to be stunned for extended periods of time.
Timeouts and Heartbeat Interval
From Aerospike side, to ensure the cluster does not split during a stun period, or if the network latency is impacted, heartbeat timeout and heartbeat interval can be adjusted to suit.
Aerospike client policies
Aerospike client applications can configure certain policies to ensure operation during stun periods. Some of the policies are outlined below.
The client applications can be configured to timeout a transaction using the socketTimeout policy and retry the transaction, up to a maximum of maxRetries or until totalTimeout is reached, whichever comes first. This allows the client application to timeout a transaction when a node is stunned in a timely manner, and retry afterward. A further parameter exists to help configure the retries, sleepBetweenRetries, which allows clients to have a configured wait period before attempting a retry. Reads can be retried immediately against a replica (this too can be configured in the read policy). Writes have to be attempted against the master node at all times. In this case, a retry can be configured, although, for writes that have succeeded, where a client did not receive a confirmation, the write will be retried. This will result in the record being written twice and its generation to increase. More details can be found in the following KB article.
Best practices
Ensure that the deployment best practices guide is followed to ensure VM and Aerospike stability.
Conclusion
Aerospike can be configured to run successfully and perform well with low latencies on VMware. Like any virtual deployment, certain tuning and considerations have to be taken into account while planning for configuration. While performance will be increased by disabling vMotion and Snapshots, it is possible to run Aerospike with the features enabled, acknowledging their usage drawbacks.
Keywords
AEROSPIKE VM VIRTUAL VMWARE VMOTION SNAPSHOT VSPHERE
Timestamp
March 2021