Bulk/Batch Updates (AER-6499)

In the end you’re always going to be limited by the aggregate write capacity of the SSDs in your cluster. You didn’t mention anything about your network cards, so that’s another physical capacity barriers your nodes may have.

To be clear, Aerospike already batches your records into larger blocks, determined on your write-block-size, and stores them onto your SSD block device. Your records are not being persisted one at a time.

It is unlikely that the Aerospike client is going to be the thing slowing you down. During your bulk update period you can do the following:

  • Modify the write-commit level policy from all to master.
  • Use the async writes of the Java client (see JavaDoc) or the C client’s regular or pipeline async writes to save on network latency.
  • Again, you will most likely be bottlenecked by the max write capacity of your SSD, not the client. if you start seeing ‘queue too deep’ warnings on your cluster nodes, and later device overload (error code 18) errors, you will probably want to dynamically increase the max-write-cache of your namespace for this update, and later reduce it to its original size.