How To use rw-client logging to identify a hot digest?

How To use rw-client logging to identify a hot digest?

Context

A hotkey is defined as a key that is being accessed in a disproportionate manner in comparison to the norm for the dataset. Hotkeys can be problematic as they can create a point load on the cluster which normally uses a balanced workload to extract maximum efficiency. The hotkey itself must be solved at the application level, however, it is possible to use the rw-client logging context to identify which digests are being accessed. How is the output from this logging context actually used.

Method

Setting the rw-client logging to detail will log every single digest accessed by clients into the aerospike.log of the node. This creates a lot of data. Once the testing for the hotfix has been completed the following command should be run against the aerospike.log

cat aerospike.log|grep rw-client|grep digest|sed -e 's/^.*digest//g'|tr ' ' ':'|awk 'BEGIN { FS = ":" } ;{print $2,$4}'|sort|uniq -c|sort -nr

The output would look as follows:

benbates@MBP: ~/Downloads % cat aerospike(2).log|grep rw-client|grep digest|sed -e 's/^.*digest//g'|tr ' ' ':'|awk 'BEGIN { FS = ":" } ;{print $2,$4}'|sort|uniq -c|sort -nr | more 
 193 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.180.12
 186 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.180.229
 115 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.181.35
  77 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.178.219
  69 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.177.72
  64 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.177.162
  62 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.176.118
  61 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.176.104
  60 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.177.15
  58 77d22f0f7620c0cf97aacef92460784f711f5d95 11.118.176.142
  14 90127ff46818c064e9fb1e0a52327c3070a2ee61 11.118.180.229
  ...
  ...
   1 f60e2776113d79ddd86091c2379350a7d8223b1a 11.118.177.162
   1 f60e2776113d79ddd86091c2379350a7d8223b1a 11.118.176.118
   1 f60e2776113d79ddd86091c2379350a7d8223b1a 11.118.176.104

From the results above it is clear that 77d22f0f7620c0cf97aacef92460784f711f5d95 is the hottest digest. This data can now be shared with the application team.

Notes

  • The rw-client logging context is detailed in the Aerospike Log Reference
  • Details on error 14 (hotkeys) messages can be found here.
  • Details on looking for read hotkeys can be found here.
  • The rw-client context can generate a lot of output when set to detail. It should be switched back to info as soon as investigations are completed.

Keywords

HOTKEY RW-CLIENT DIGEST LOGGING

Timestamp

May 2021

© 2021 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.