XDR causing transactions pending in queue threshold warning


#1

FAQ - Transactions pending in queue threshold warning on AMC while XDR is recovering from high lag

Detail

XDR is very efficient, and if its throughput is not limited (by default it is not), it will ship as much as it can to the destination DCs. This can lead to a certain jump in throughput if XDR had a lot of outstanding records digest in the digestlog. (The high backlog could be caused by a number of factors including a high lag, a DC being down, shipping being disabled temporarily, etc.) Upon starting XDR and resuming shipping with a high backlog of record outstanding, the following error can be triggered on Aerospike Monitoring console(AMC).

WARNING: Transactions pending in queue greater than 10000 !

Answer

The above warning is triggered by a high number of transactions being added to the transactions queue due to the high rate of reading and shipping records that had their digest outstanding in the digestlog. This error should subside once all the records have been shipped and transaction queue should go back to previous levels.

The high rate of shipping through XDR can be throttled using the following setting.

http://www.aerospike.com/docs/reference/configuration#xdr-max-ship-throughput

Notes

The logs can be parsed for the following keywords:

  • dlog-outstanding : number of outstanding records should decrease with a high rate (corresponds to the xdr_ship_outstanding_objects statistic).
  • tsvc-q : The transaction queue value will show a number greater than 10000 and should also decrease over time as objects are shipped (corresponds to the tsvc_queue statistic).

Keywords

XDR BACKLOG OUTSTANDING tsvc-q

Timestamp

02/17/17