In our application , we are doing batch reads in a batch of 90. We are observing that when we do “show latencies” in asadm mode , we are getting around 85K Qps consistently for around 30 mins.
However , when we checked average increase in batch_sub_read_success metrics , it is around 185K. Not able to understand even when traffic is staying constant , then how come average rate of increase of batch_sub_read_success is not same as real time ops/seconds response of show latencies. Also , almost all reads are success , so just ignore failures/exceptions/time-outs for now.
Am I missing something?
Version: 6.1 EE.
The show latencies command would not show you the number of read transactions in a batch. It would show you the number of batch transactions. The batch_sub_read metric would show you the number of individual batch sub transactions and taking the QPS of batch_sub_read divided by the QPS of batch (batch_index_complete – potentially including the related ones for not found or filtered out or even errors) would give you an idea of the number of read transactions per batch. Increasing the number of read transactions per batch I don’t think wouldn’t show up in show latencies until version 6.4 (unless if enabling the batch histogram benchmarks).
Feel free to share the outputs of what you are seeing but as you are running EE, directly open a case with Support.
Thanks Meher. I understood from your reply that then batch_sub_read_success is the correct metric to monitor in our case then for real time qps.
But my primary doubt still remains that if all operations are batch operations only, then result of show latencies (ops/sec column) , shouldn’t it match with rate of increase of batch_sub_read_success? What causes the deviation?
As per my understanding , consider two application side get requests of 90 reads each on after the other.
Does the Java client not send these 180 requests separately or does it combine some of them in batches for network improvement etc.?
If you have two applications sending each a batch of 90 requests, each node in the cluster will receive two batch requests. If a single node in the cluster, then the node will receive two batches and that will be what shows up under the show latencies and the node will receive all 180 batch sub read transactions (if you had a two node cluster, it would be roughly half for each, etc…). In version 6.4 you should also see the batch sub transactions (see {ns}-batch-sub-read).
Basically a batch transaction shows up as a single transaction and the individual read transactions show up as sub transactions and not represented on show latencies (well until version 6.4).