Could you share your namespace configuration? Also could look into jemstats.
(appears the kb for jemstats isn’t finished - following is an excerpt of the draft)
[AER-5683] - (KVS) New memory subsystem with configurable allocation debugging.
This is what memory debugging offers:
- Memory accounting: We keep track of which code location has allocated how much memory. This helps in the case of memory leaks.
- Double-free detection and corruption detection: This helps us find issues with Aerospike code, where there is potentially improper manage allocations or some parts of the code making different assumptions about the allocations than other parts causing unexpected bugs.
There are two ways to interact with the memory subsystem as an Aerospike user.
By default, this is disabled. Enabling this would assert on double frees and corruption detections which otherwise could go unnoticed. There is a potential risk when enabling this.
The configuration in the
service stanza of the Aerospike configuration file and is a
static configuration (requires service restart).
It takes a one of the following four values as a parameter:
none - This completely disables any instrumentation of the allocation API: memory accounting, buffer overflow detection, double free detection. Instead, we simply forward any API calls directly to JEMalloc. In particular, this removes our 4-byte memory overhead per allocation.
all - This enables instrumentation for all allocations and, thus, incurs the 4-byte memory overhead for all allocations.
transient - This enables instrumentation only for transient - i.e., short-lived - allocations. Technically speaking, this exempts allocations from namespace arenas, whereas allocations from a thread’s default arena are covered by the instrumentation.
persistent - This is the complement to
transient. This setting enables instrumentation for allocations from namespace arenas, whereas allocations from a thread’s default arena are exempt.
jem-stats asinfo command:
This is used to dump JEMalloc’s internal statistics as well as our own memory accounting information, i.e., our site_info records. Analyzing the output does require scripting using Python.
asinfo -v 'jem-stats:file=[...];options=[...];sites=[...]'
The first two options to the info command concern JEMalloc’s internal statistics:
file option specifies the file to receive JEMalloc’s internal statistics. The internal statistics are dumped via JEMalloc’s jem_malloc_stats_print(). If no file option is specified, the internal statistics are dumped to the Aerospike log file.
options option is simply forwarded to the options parameter of jem_malloc_stats_print().
If specified, the
sites option indicates that we would also like to dump the site_info records for all threads and all allocation sites. The option specifies the file to which to dump this information. The resulting file contains lines like the following, each of which describes one site_info record.
0x00000000005126c4 8762 0x0000000000000000 0x0000000000000240
0x00000000005126c4 8763 0xffffffffffffffff 0xfffffffffffffdc0
The four columns have the following meaning:
- The code address of the allocation site of the site_info record.
- The thread ID of the thread owning the site_info record.
- The upper 64 bits of the 128-bit size in the site_info record (size_hi member).
- The lower 64 bits of the size (size_lo member).
In the above example, we’re looking at the following allocation site:
$ addr2line -e /usr/bin/asd 0x5126c4
It looks like one thread, thread 8762, allocated 0x240 = 576 bytes at system_metadata.c:729. And another thread, thread 8763, seems to have deallocated these 576 bytes, resulting in a size member of -576 = 0xffff…fdc0.
Note: Further processing of the dumped raw site_info data would typically be done with, say, a Python script. At this stage, Aerospike daemon itself doesn’t contain any facilities for further analysis.
Sample script: There is a Aerospike developed Python script that does this which is available packaged with Aerospike Tools package.
Usage: To translate “input-file” (which was obtained via “
jem-status:” and which has memory addresses) into “
output-file” (which has human-readable file names and line numbers).
asparsemem -a /usr/bin/asd -i input-file -o output-file