The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
AMC Is Now Community Supported!
Aerospike no longer supports the AMC, which has been replaced by the Aerospike Monitoring Stack. The AMC has been turned over to the community, with code hosted at aerospike-community/amc and documentation on the repo Wiki.
FAQ - Transactions pending in queue threshold warning on AMC while XDR is recovering from high lag
Detail
XDR is very efficient, and if its throughput is not limited (by default it is not), it will ship as much as it can to the destination DCs. This can lead to a certain jump in throughput if XDR had a lot of outstanding records digest in the digestlog. (The high backlog could be caused by a number of factors including a high lag, a DC being down, shipping being disabled temporarily, etc.) Upon starting XDR and resuming shipping with a high backlog of record outstanding, the following error can be triggered on Aerospike Monitoring console(AMC).
WARNING: Transactions pending in queue greater than 10000 !
Answer
The above warning is triggered by a high number of transactions being added to the transactions queue due to the high rate of reading and shipping records that had their digest outstanding in the digestlog. This error should subside once all the records have been shipped and transaction queue should go back to previous levels.
The high rate of shipping through XDR can be throttled using the following setting.
Notes
The logs can be parsed for the following keywords:
- dlog-outstanding : number of outstanding records should decrease with a high rate (corresponds to the xdr_ship_outstanding_objects statistic).
- tsvc-q : The transaction queue value will show a number greater than 10000 and should also decrease over time as objects are shipped (corresponds to the tsvc_queue statistic).
Applies To
Server prior to v. 5.0
Keywords
XDR BACKLOG OUTSTANDING tsvc-q
Timestamp
02/17/17