I am encountering this error during operation.
“All batch queues are full” aerospike exception
I am performing large batch reads and I presume it caused this error.
It will be good if someone can share the mechanics of batch reads in Aerospike in more detail and if possible some hints of avoiding them.
Thank you!
What version of Aerospike are you using? Batch requests have been improved in the 3.6.0 release.
The version I am running is 3.6.0. I have to admit that the number of batch reads I am running is very very large though. I am running aerospike against a storm cluster so numerous bolts will be running against the db.
Ok so you are exceeding batch-max-buffer-per-queue for all batch-index-threads. I think we should try increasing the number concurrent batch requests by increasing batch-index-threads, by default it is 4. A good value would probably be the number of cores on this box.
These parameters are dynamically configurable:
asadm -e "asinfo -v 'set-config:context=service;batch-index-threads=NEW_VALUE'"
# Be sure to update your static configs for future service restarts.
Also you should take a look at batch-max-unused-buffers. This controls when unused 128 KiB buffers will be garbage collected and by default is 256. You will want to tune this to a level where your normal load isn’t constantly garbage collecting. See batch_index_queue, I think it would be helpful here.
asadm -e "show stat like batch_index"
# batch-max-unused-buffers can be configured dynamically with:
asadm -e "asinfo -v 'set-config:context=service;batch-max-unused-buffers=NEW_VALUE'"
# Again be sure to update your static config.