I am using multiple ENIs on my ec2 node, how can i configure aerospike to use all the ENIs?
I am using the following setting in network section, is this enough?
service { address any port 3000 }
Also, should i mention the access-address explicitly? (i don’t mind user/client connecting on any ENI)
Yes, the snippet of the config that you shared is good for making aerospike listen on all the ENIs.
May I know why you are using multiple ENIs ? If you are not using multiple ENIs for performance reasons, we recommend to set the access-address to one of the IPs. This will simplify the cluster discovery process by the clients.
Let us know if you have any problems with it.
Thank you. We are using multiple ENIs for performance.
Will not specifying the access-address degrade the client performance? All i need to tell the client is the IP of any node in the cluster (IP could be of any ENI), right? Or is there a best practice for it?
You should not be specifying the access-address if you have multiple ENIs for performance purpose. Not all clients are tuned to handle multiple IPs of a same node. Java client definitely does. If you are using java, you are good to go.
To exploit all the ENIs of cluster, you need to have a small hack in the client application. Lets say you have 3 boxes A, B, C with each one having 4 nics like A1, A2, A3, A4. and so on. What you should do is to specify different nics in the seed list of each client instances. i.e for client instance one give seed list as (A1, B1, C1) and for second client instance give (A2, B2, C2).
Note that this hack is needed only when you have mutliple nics and for perf reasons. If its not for perf reasons, you can simply give A1 as seed IP for all client instances and they will discover the rest of the cluster. It may not use all the NICs in this case.
Suppose I start the node with 1 ENI and later plan to add 3 more, will a simple aerospike restart be enough after adding 3 more ENIs? (in conf i have ‘address any’).
To grow the cluster in space, all i need to do is just adding servers? Also, will it get rebalanced by itself?
Yes, you can add more ENIs later and simply restart aerospike to start using them.
Adding more nodes is a common way of adding capacity to the cluster. (Adding more disks & ram is also another alternative, if you do not want to maintain more machines) Yes, the data will auto-rebalance. You need not do anything special for it.
Just a small correction - when adding more ENIs to a running aerospike server with address any in config, you do not need to restart aerospike server. As the server is listening on all interfaces, it discovers new ENIs and starts listening on them as well.
The clients also will mostly not need restart either.
Is this hack still needed? I’m using the Java client and have multiple ENIs (1 per cpu in EC2) on my cluster nodes to avoid pegging one of the 4 cpus with network interrupts. However, based on what I’ve tried with client configurations and your comment, it sounds like I need to list the IP of every ENI in the cluster in order for this work? It also means I need to have one AsyncClient per #-of-ENIS-per-node ? Currently my application is using only one AsyncClient. It’d be a bit of work to switch it to using multiple clients. Is there no way for the client to know about these other ENI/IPs via seeding?
hi andersonbd1,
Yes, unfortunately making the smart clients smarter by adding the intelligence to handle multiple unique IPs on a server is not trivial. We are discussing the way to do it properly with the team but in the meanwhile you still need the hack.
We will update this thread as and when this is part of the main client making the hack obsolete.
BTW, a simpler technique to use all the NICs of the box is to use bonding (link aggregation). Multiple NICs can be bonded together to form one logical NIC.