How To use rw-client logging to identify a hot digest?

How To use rw-client logging to identify a hot digest?


A hotkey is defined as a key that is being accessed in a disproportionate manner in comparison to the norm for the dataset. Hotkeys can be problematic as they can create a point load on the cluster which normally uses a balanced workload to extract maximum efficiency. The hotkey itself must be solved at the application level, however, it is possible to use the rw-client logging context to identify which digests are being accessed. How is the output from this logging context actually used.


Setting the rw-client logging to detail will log every single digest accessed by clients into the aerospike.log of the node. This creates a lot of data. Once the testing for the hotfix has been completed the following command should be run against the aerospike.log

cat aerospike.log|grep rw-client|grep digest|sed -e 's/^.*digest//g'|tr ' ' ':'|awk 'BEGIN { FS = ":" } ;{print $2,$4}'|sort|uniq -c|sort -nr

The output would look as follows:

benbates@MBP: ~/Downloads % cat aerospike(2).log|grep rw-client|grep digest|sed -e 's/^.*digest//g'|tr ' ' ':'|awk 'BEGIN { FS = ":" } ;{print $2,$4}'|sort|uniq -c|sort -nr | more 
 193 77d22f0f7620c0cf97aacef92460784f711f5d95
 186 77d22f0f7620c0cf97aacef92460784f711f5d95
 115 77d22f0f7620c0cf97aacef92460784f711f5d95
  77 77d22f0f7620c0cf97aacef92460784f711f5d95
  69 77d22f0f7620c0cf97aacef92460784f711f5d95
  64 77d22f0f7620c0cf97aacef92460784f711f5d95
  62 77d22f0f7620c0cf97aacef92460784f711f5d95
  61 77d22f0f7620c0cf97aacef92460784f711f5d95
  60 77d22f0f7620c0cf97aacef92460784f711f5d95
  58 77d22f0f7620c0cf97aacef92460784f711f5d95
  14 90127ff46818c064e9fb1e0a52327c3070a2ee61
   1 f60e2776113d79ddd86091c2379350a7d8223b1a
   1 f60e2776113d79ddd86091c2379350a7d8223b1a
   1 f60e2776113d79ddd86091c2379350a7d8223b1a

From the results above it is clear that 77d22f0f7620c0cf97aacef92460784f711f5d95 is the hottest digest. This data can now be shared with the application team.


  • The rw-client logging context is detailed in the Aerospike Log Reference
  • Details on error 14 (hotkeys) messages can be found here.
  • Details on looking for read hotkeys can be found here.
  • The rw-client context can generate a lot of output when set to detail. It should be switched back to info as soon as investigations are completed.




May 2021

© 2021 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.