How do I estimate the XDR throughput?

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Context

I am trying to figure out how many records is XDR shipping. How do I calculate it based on server logs and statistics?

Method

To calculate the number of keys that are attempted to be shipped (this includes the ones that can potentially error out), we need to monitor the number after sh. Note that this represents the number of keys shipped across all DCs.

For a single-destination:

Nov 14 2017 23:45:04 GMT: INFO (xdr): (xdr.c:628) summary: throughput 50 inflight 0 dlog-outstanding 0 dlog-delta-per-sec 0.0
Nov 14 2017 23:45:04 GMT: INFO (xdr): (xdr.c:638) detail: sh 4718785 ul 53 lg 9844488 rlg 0 rlgi 2 rlgo 0 lproc 9844488 rproc 808307 lkdproc 0 errcl 0 errsrv 0 hkskip 2059 hkf 2054 flat 0
Nov 14 2017 23:45:14 GMT: INFO (xdr): (xdr.c:628) summary: throughput 50 inflight 0 dlog-outstanding 0 dlog-delta-per-sec 0.0
Nov 14 2017 23:45:14 GMT: INFO (xdr): (xdr.c:638) detail: sh 4719289 ul 52 lg 9845521 rlg 0 rlgi 2 rlgo 0 lproc 9845521 rproc 808307 lkdproc 0 errcl 0 errsrv 0 hkskip 2059 hkf 2054 flat 0

From asadm, you can monitor using the following:

Admin> info xdr
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~XDR Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Node      Build         Data    Free         Lag           Req       Req       Req          Cur       Avg   
                   .          .      Shipped   Dlog%       (sec)   Outstanding   Shipped   Shipped   Throughput   Latency   
                   .          .            .       .           .             .   Success    Errors            .      (ms)   
192.168.100.207:3000   3.15.0.2   596.244 MB       1   00:00:00        3.000     4.718 M   0.000             49         1  
Number of rows: 1

The throughput (also available directly in the logs after throughput) can be calculated as well by subtracting the total shipped records and dividing by the ticker interval, in this example:

(4719289 - 4718785) / 10 = 50.4

Using raw statistics, what you need is the following:

sh =  xdr_ship_success + xdr_ship_source_error + xdr_ship_destination_error

Notes

  • sh value does take in account if the cluster has multiple destinations.
  • sh takes into account the hot key fetches that were done if you do have XDR hotkeys.

Applies To

Server prior to v. 5.0

Keywords

XDR throughput calculation estimate

Timestamp

11/14/2017