How do I estimate the XDR throughput?


#1

Context

I am trying to figure out how many records is XDR shipping. How do I calculate it based on server logs and statistics?

Method

To calculate the number of keys that are attempted to be shipped (this includes the ones that can potentially error out), we need to monitor the number after sh. Note that this represents the number of keys shipped across all DCs.

For a single-destination:

Nov 14 2017 23:45:04 GMT: INFO (xdr): (xdr.c:628) summary: throughput 50 inflight 0 dlog-outstanding 0 dlog-delta-per-sec 0.0
Nov 14 2017 23:45:04 GMT: INFO (xdr): (xdr.c:638) detail: sh 4718785 ul 53 lg 9844488 rlg 0 rlgi 2 rlgo 0 lproc 9844488 rproc 808307 lkdproc 0 errcl 0 errsrv 0 hkskip 2059 hkf 2054 flat 0
Nov 14 2017 23:45:14 GMT: INFO (xdr): (xdr.c:628) summary: throughput 50 inflight 0 dlog-outstanding 0 dlog-delta-per-sec 0.0
Nov 14 2017 23:45:14 GMT: INFO (xdr): (xdr.c:638) detail: sh 4719289 ul 52 lg 9845521 rlg 0 rlgi 2 rlgo 0 lproc 9845521 rproc 808307 lkdproc 0 errcl 0 errsrv 0 hkskip 2059 hkf 2054 flat 0

From asadm, you can monitor using the following:

Admin> info xdr
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~XDR Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Node      Build         Data    Free         Lag           Req       Req       Req          Cur       Avg   
                   .          .      Shipped   Dlog%       (sec)   Outstanding   Shipped   Shipped   Throughput   Latency   
                   .          .            .       .           .             .   Success    Errors            .      (ms)   
192.168.100.207:3000   3.15.0.2   596.244 MB       1   00:00:00        3.000     4.718 M   0.000             49         1  
Number of rows: 1

The throughput (also available directly in the logs after throughput) can be calculated as well by subtracting the total shipped records and dividing by the ticker interval, in this example:

(4719289 - 4718785) / 10 = 50.4

Using raw statistics, what you need is the following: https://www.aerospike.com/docs/reference/metrics#xdr_ship_success https://www.aerospike.com/docs/reference/metrics#xdr_ship_source_error https://www.aerospike.com/docs/reference/metrics#xdr_ship_destination_error

sh =  xdr_ship_success + xdr_ship_source_error + xdr_ship_destination_error

Notes

  • sh value does take in account if the cluster has multiple destinations.
  • sh takes into account the hot key fetches that were done if you do have XDR hotkeys.

Keywords

XDR throughput calculation estimate

Timestamp

11/14/2017