Aerospike performance with mesh clustering on AWS EC2

Hi,

I’m trying to experiment with Aerospike. Below are what I did:

  1. Run aerospike on an EC2 instance, then run the benchmark tool on another instance. From the benchmark tool log I can see the tps hovers around 200k tps.
  2. Setup another aerospike server on another EC2 instance, and setup mesh clustering mode. Now from the benchmark tool log I see the tps drops down to around 100k tps.
  3. Setup another aerospike server on a third instance, adding it to the cluster. The benchmark tool still shows about 100k tps.
  4. Setup another benchmark client to send requests to the 2nd server. Now both benchmark results drop down to about 50k tps.

All the benchmark was run with

-p 3000 -n test -k 10000000 -S 1 -o S:100 -w RU,50 -z 90 -latency 10,1

My question is why does the performance drop so much? Is it expected? Is it because it’ll take time to sync the data across all the instances? What can I do to increase the performance of the cluster? Would changing service-threads and transaction-queues help?

Ideally I’d expect the benchmark tool result to be at least the same as when running on a single instance.

Below are the heartbeat setup part for the 3 servers:

heartbeat { mode mesh port 3002 address 10.00.00.01 mesh-seed-address-port 10.00.00.01 3002 mesh-seed-address-port 10.00.00.02 3002 mesh-seed-address-port 10.00.00.03 3002 interval 150 timeout 10 }

heartbeat { mode mesh port 3002 address 10.00.00.02 mesh-seed-address-port 10.00.00.01 3002 mesh-seed-address-port 10.00.00.02 3002 mesh-seed-address-port 10.00.00.03 3002 interval 150 timeout 10 }

heartbeat { mode mesh port 3002 address 10.00.00.03 mesh-seed-address-port 10.00.00.01 3002 mesh-seed-address-port 10.00.00.02 3002 mesh-seed-address-port 10.00.00.03 3002 interval 150 timeout 10 }

Please add the full config.