XDR and shipping of hotkeys - xdr-hotkey-maxskip


#1

Synopsis

When tuning XDR its best practice to adjust xdr-hotkey-maxskip and reduce the number of updates sent over the wire for a particular hotkey.

Understanding when to skip shipping intermediary updates

A good example to understand the need to skip shipping intermediary updates when only the final update matters would be the database for a voting application with two candidates. If for example only two keys are being updated for counting the number of votes for a candidate. The last updates have the final increments. Sending over the wire all the intermediary updates could be considered as wasting your XDR shipping bandwidth. Adjusting xdr-hotkey-maxskip allows us to skip a certain number of record generation numbers for a record in the digest log for a “Hot” Key.

In order to determine if you have a hotkey situation one should check at what speed XDR is processing records per second and compare this rate to noship_recs_mismatch rate. if noship_recs_mismatch is increasing at the same rate you will have the same record that is being updated.

A simple thumb rule for determining a value for xdr-hotkey-maxskip

When the lag is less than 10 secs, we recommend xdr-hotkey-maxskip be set to a value that is less than 100. If we are in a high XDR shipping lag situation, we can increase the xdr-hotkey-maxskip setting to a larger number. Setting xdr-hotkey-maxskip to N will not skip all the N-1 values. XDR will skip N-1 records OR skip up to the latest record with matching generation between digest log and actual record, whichever happens earlier. (for e.g if xdr-hotkey-maxskip is set to 100 and record generation is 20 and then lets say the record is updated in quick succession 10 times, its generations will become 30 before XDR ships it. When XDR considers the record for shipping, the generation in the log and the actual record will match. It will ship without waiting for the generation to become 100.)

Command syntax using asinfo :

Retrieving current setting:

asinfo -p 3004 -v 'get-config:' -l|grep maxskip

Changing setting dynamically:

asinfo -p 3004 -v 'set-config:context=xdr;xdr-hotkey-maxskip=1000'

Note

Permanently change the setting in aerospike.conf by adding the setting to the XDR stanza.