FAQ - What is the max size of a records primary key in Aeropsike


#1

FAQ - What is the max size of a records primary key in Aeropsike.

Details

Aerospike can handle strings or numeric values as its primary key. The original key will be hashed (RipeMD160) to generate a unique record identifier known as the digest.(See http://www.aerospike.com/docs/architecture/data-distribution.html)

Answer

The original key is not stored in the database by default. Aeropsike will only store the 20 bytes digest hash of the original key as well as the set name (if specified). This would imply that the original key could be of any size that the Client can handle. There are however some limit. Let’s first look at the Java client as an example.

The java client limits the key to slightly under 8 KB. Here is the reason:

  • The computeDigest() method is used to create a 20 byte hash digest from a set name, user key type, and variable sized user key. This 20 byte digest will be sent to the server. The user key will not be sent to the server by default thus implying there is no limit.

  • The computeDigest() method uses a temporary thread local buffer initially sized at 8 KB to assemble bytes for use with the RipeMD160 hashing method.

  • This temporary buffer size is the defacto limit in the java client.

For C based client: (python, php, C)

No limit defined. Actual limit is the size of the stack at around 10MB.

The stack size can be determined by running:

ulimit -s
10240

or

ulimit -a

[vagrant@localhost ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7424
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[vagrant@localhost ~]$ ulimit -s
10240

Notes

  • It would have of course been possible to add code to estimate the size and dynamically create a buffer in the Java client thus allowing keys of any size, but this would impact performance. In general, 8KB should be a very reasonable limit for the vast majority of use cases.

Keywords

Max userKey size client limit

Timestamp

12/15/2016