I am trying to load a 8 GB file into an empty Aerospike namespace.
Here are the options I use:
-h <one host's IP> -p 3000 -c ~/config.json -n pc ~/data.dsv
I notice that when I load the file into a single node cluster on AWS, the asloader logged TPS of 130,000. However when I loaded the same file, with same options, in a 4 node cluster on AWS (4 nodes in a region, with 2 nodes per AZ), the asloader logs a TPS of just 35,000.
Is this because of the writes running in Strong Consistency mode, with each batch being committed in the replica? How can I make this faster?
Assuming you have a replication factor (RF) > 1 in the multi-node cluster, the writes are written to all replicas. This will increase the write latency and reduce the throughput (this is irrespective of the SC/AP mode in a normal connected cluster). Other factors contributing to the slowdown can be slower connection speed between the two AZs or the client and an AZ; or a slower node(s) or devices. You may want to run the single node test with the second AZ to see if the throughput remains the same. Typically higher device and network speeds will get your better write throughput.
Side note: you could also leverage (in AP mode only – not allowed in Strong Consistency) fire and forget if you don’t mind potential write replication failures that wouldn’t be reported to the client. Refer to the write-commit-level-override configuration parameter. This would allow higher throughput even with replication factor greater than 1.