How to interpret Aerospike jemalloc stats data

How to interpret Aerospike jemalloc stats data

Context

jemalloc is the sub-system used by Aerospike to manage memory allocations. Sometimes there may be a requirement for Aerospike Support to look at jemalloc in detail. This is often in cases where unexpected memory usage is noticed. There is a command that can be used to dump information on jemalloc into flat files. It is as follows:

asinfo -v 'jem-stats:file=/tmp/jem.txt;sites=/tmp/sites.txt'

The purpose of this article is to attempt to explain what information is being shown in the jem.txt file. In most circumstances, it is necessary to engage core developers to make a full analysis of the jemalloc sub-system and any associated issues.

Detail

In general, jemalloc requests memory from the underlying operating system in allocations of 2 MiB. A section of a jem.txt file is shown below and the size of this allocation is listed as Chunk size.

---------- Feb 12 2021 18:16:57 GMT ----------
___ Begin jemalloc statistics ___
Version: 4.2.1-3-g05108b5010a511226fb7586543f4162dd2d31d2b
Assertions disabled
config.malloc_conf: ""
Run-time option settings:
  opt.abort: false
  opt.lg_chunk: 21
  opt.dss: "secondary"
  opt.narenas: 149
  opt.purge: "ratio"
  opt.lg_dirty_mult: 3 (arenas.lg_dirty_mult: 3)
  opt.stats_print: false
  opt.junk: "false"
  opt.quarantine: 0
  opt.redzone: false
  opt.zero: false
  opt.tcache: true
  opt.lg_tcache_max: 15
CPUs: 16
Arenas: 151
Pointer size: 8
Quantum size: 16
Page size: 4096
Min active:dirty page ratio per arena: 8:1
Maximum thread-cached size class: 32768
Chunk size: 2097152 (2^21)
Allocated: 70421102784, active: 70674239488, metadata: 2874128256, resident: 74950987776, mapped: 112502243328, retained: 910163968
Current active ceiling: 70837600256

Another interesting piece of information to note is the Page size of 4096 bytes or 4 KiB. This is why the Linux Transparent Huge Page setting causes issues with Aerospike. jemalloc expects a 4 KiB page and so if it is given a 2 MiB allocation it will not release this correctly. This can lead to behaviour with similar symptoms to a memory leak.

jemalloc summarises allocations requested by ASD into 3 broad groupings, small, large, and huge. As an allocation is requested, jemalloc classifies the request into one of those 3 categories. This information is also shown in the jem.txt file in the next section down. Small allocations are smaller than the page size, large allocations are smaller than the chunk size.

Merged arenas stats:
assigned threads: 153
dss allocation precedence: N/A
min active:dirty page ratio: N/A
purging: dirty: 343856, sweeps: 16350, madvises: 31453, purged: 4968403
                            allocated      nmalloc      ndalloc    nrequests
small:                       24912064  11881798136  11881747920  21077736082
large:                    70055927808   1348157773   1348103516  14954312690
huge:                       340262912        25578        25487        25578
total:                    70421102784  13229981487  13229876923  36032074350

As the title line above states, this is the summary of allocations for all arenas, an arena being a discrete area of memory. jemalloc allows the independent allocation of memory across arenas. The reason for this is that the architecture is constrained in that an arena can only be used by a single thread at any given time. Arenas, therefore, have to work independently as otherwise there would be contention as threads blocked each other and the system would run extremely slowly.

In addition to the high-level summary shown above, the jem.txt file will list detailed information around the allocations being made within the various arenas. This is shown below.

bins:           size ind    allocated      nmalloc      ndalloc    nrequests      curregs      curruns regs pgs  util       nfills     nflushes      newruns       reruns
                   8   0         3168          406           10          202          396            4  512   1 0.193            4        52463            4            0
                  16   1        11536         1116          395          219          721           11  256   1 0.256           27        52494           12            0
                  32   2       257216      7395267      7387229     11603893         8038          141  128   1 0.445       542934       612121        11069        97165
                  48   3       129744     14305553     14302850      8860415         2703           88  256   3 0.119      1086717      2473841        42029            0
                  64   4       416320    757727955    757721450    777714403         6505          260   64   1 0.390     11959726     12866311        15604     24938301
                  80   5        29200       497356       496991       222478          365           24  256   5 0.059       134976       228802        26594            0
                  96   6        24000        12089        11839         1524          250            4  128   3 0.488         1210        55209         1206            0
                 112   7       662928    292758391    292752472    292661003         5919           82  256   7 0.281      3081699      3120768         9120            0
                 128   8        62720    303125691    303125201    499267792          490           92   32   1 0.166      9757642     10094553       339072     19512919
                 160   9       150080    754866404    754865466   1167044228          938           79  128   5 0.092      8105325      8846606      1323643      9313340
                 192  10        43584    519061169    519060942    885767584          227           68   64   3 0.052      8694790      9350166       855897     16066372
                 224  11        60256    369416805    369416536    548717466          269           18  128   7 0.116      4183320      4965265       664647      3470672
                 256  12        13824     34908819     34908765     45109533           54           10   16   1 0.337      2525229      3379724       808502      3428112
                 320  13      1303680    260169288    260165214    285536086         4074          163   64   5 0.390      4526382      5657328       661003      7280658
                 384  14      1082112    736421887    736419069    969147096         2818          196   32   3 0.449     23326020     24094398      3270901     43713028
                 448  15      1784384    678934012    678930029   1230557105         3983          169   64   7 0.368     10772748     11491226       287105     20912455
                 512  16       291840    936007040    936006470   1126962476          570          167    8   1 0.426     93839366     94409388     66123133    139897634
                 640  17      1305600   3287842194   3287840154   8999168268         2040          168   32   5 0.379    104778844     13062446     24507316    176939811
                 768  18       764928    112128783    112127787    407304951          996          156   16   3 0.399      7378414      8214609      1801376     12141522
                 896  19      1654912     40328540     40326693    163019683         1847          131   32   7 0.440      1700600      2947741       353651      1532065
                1024  20       590848     45069200     45068623    198881577          577          230    4   1 0.627      4776223      5592083      8803862     10818362
                1280  21      4427520     47187040     47183581    290971957         3459          354   16   5 0.610      3235099      4210903       113626      5970100
                1536  22       761856     18548306     18547810     63844330          496          132    8   3 0.469      2195639      2955876      1447507      2291860
                1792  23       906752     11877761     11877255     30294076          506           90   16   7 0.351      1255151      2284750       556720       472337
                2048  24       579584      7472660      7472377     19370026          283          176    2   1 0.803      1196500      2063952      3533730      1746838
                2560  25      1029120      8084121      8083719     25557094          402          123    8   5 0.408      1196507      1940468       702396       679219
                3072  26       313344      5116616      5116514     15291716          102           56    4   3 0.455       864157      1613699      1155222       958312
                3584  27       318976      3525326      3525237      9424028           89           40    8   7 0.278       700106      1448441       450362       135598
                4096  28       704512      2646947      2646775      5801014          172          172    1   1 1           613521      1299744      2646947            0
                5120  29      3875840   2617109581   2617108824   2696148170          757          308    4   5 0.614    262051614    262495517    465126795    706628920
                6144  30       516096      2573396      2573312    296331849           84           74    2   3 0.567       615748      1220848      1257495       546953
                7168  31       157696      1710041      1710019      2024526           22            8    4   7 0.687       443326       949211       589851       198717
                8192  32       163840      1374388      1374368      1399435           20           20    1   2 1           378922       819733      1374388            0
               10240  33       276480      1613239      1613212      1827013           27           15    2   5 0.900       381638       847271       822805       280997
               12288  34        36864      1132391      1132388      1137048            3            3    1   3 1           276760       662270      1132391            0
               14336  35       200704       848358       848344       765818           14            8    2   7 0.875       228190       543725       454888       100076
large:          size ind    allocated      nmalloc      ndalloc    nrequests      curruns
               16384  36      1114112       312593       312525  13602574671           68
               20480  37        81920       624833       624829      3104745            4
               24576  38       147456       604905       604899      1515978            6
               28672  39      4472832       548347       548191       910047          156
               32768  40        32768       364804       364803       504958            1
               40960  41       122880    149915167    149915164    149915167            3
               49152  42        49152       193042       193041       193042            1
               57344  43            0       123609       123609       123609            0
               65536  44       262144        90494        90490        90494            4
               81920  45       573440   1194740467   1194740460   1194740467            7
               98304  46      9830400       265718       265618       265718          100
              114688  47            0        16437        16437        16437            0
              131072  48       131072        14913        14912        14913            1
              163840  49     39976960         4481         4237         4481          244
              196608  50            0          197          197          197            0
              229376  51            0          222          222          222            0
              262144  52            0          165          165          165            0
              327680  53     29491200       279712       279622       279712           90
              393216  54            0            8            8            8            0
              458752  55       917504            4            2            4            2
              524288  56            0            1            1            1            0
              655360  57    298188800         1357          902         1357          455
                     ---
              917504  59            0         2557         2557         2557            0
             1048576  60            0            1            1            1            0
             1310720  61  69360680960        53542          624        53542        52918
             1572864  62    309854208          197            0          197          197
                     ---
huge:           size ind    allocated      nmalloc      ndalloc    nrequests   curhchunks
             2097152  64      2097152            2            1            2            1
             2621440  65    212336640        25563        25482        25563           81
                     ---
             4194304  68            0            1            1            1            0
                     ---
             8388608  72            0            1            1            1            0
            10485760  73     83886080            9            1            9            8
                     ---
            16777216  76            0            1            1            1            0
                     ---
            41943040  81     41943040            1            0            1            1
                     ---

The information above is similar in nature to a histogram. The bins column, on the left-hand side, is defining the width of a bin. The width of a bin in bytes (end of previous bin + 1) to (figure quoted). So the size of the second bin is from 9 bytes to 16 bytes. The allocated column shows how many allocations, of size within that bucket range, are currently active. In the information above there have been 12 allocations within the 225 bytes to 256-byte range.

For clarity, the buckets are split into their small, large, and huge groupings. This also defines the size of those groupings. The huge category runs from 1835009 bytes to 41943040 bytes, for example.

One technique, when tracking down unexpected memory usage, would be to look at the number of allocations within each bucket and look for a number of allocations that is disproportionately large. This could indicate an unexpected allocation or even, in rare cases, an allocation being made but never released.

After the summary information for all arenas, jem.txt reports allocation by arena in the same form. By default, Aerospike creates 150 arenas. Additional arenas may be created if data is stored in memory. In the example above there are 151 arenas and so at least one namespace is configured to store data in memory.

All threads being used by the asd process are assigned to arenas randomly. It is possible to have more than one thread assigned to an arena. Even with large numbers of threads configured (such as an XDR enabled cluster would have), across 150 arenas, there is still minimal contention and thus high speed.

In terms of how this information can be used, after finding a bucket where allocations are disproportionately high in the merged arena data, that bucket is checked in individual arenas. If a single arena contains the bulk of the allocations and if that arena has a single thread then it could be inferred that a single thread sub-system of asd was responsible for the allocations. This would be something like NSUP or the thread responsible for accepting client connections.

If there is no such direct relation to be found, the sites.txt file is used. An abbreviated example is shown below.

---------- Feb 12 2021 18:16:57 GMT ----------
0x00007f2d7dab856d    126646 0x0000000000000000 0x0000000000000000
0x00007f2d7dab7aec    126646 0x0000000000000000 0x0000000000000000
0x000000000045c78f    126646 0x0000000000000000 0x00000000000001c0

The first column is a 12-bit site ID which denotes a location in which asd is allocating memory. The second column is a thread ID for the thread making the allocation and the final two columns give the value of the allocation. This would be used to determine which exact thread was making the allocations and for what reason.

There is an info command thread-traces that can be run. The output will show the thread ID, the offset of its run function and its current call stack. Example output is shown below.

asinfo -v 'thread-traces' -l
---------- 45387 (0x12914f) ----------
/usr/bin/asd(cf_thread_traces_action+0x4c) [0x67807c]
/lib64/libpthread.so.0() [0x384720f710]
/lib64/libc.so.6(nanosleep+0x2d) [0x3846eaca3d]
/lib64/libc.so.6(usleep+0x34) [0x3846ee1be4]
/usr/bin/asd(hb_adjacency_tender+0x275) [0x5293c4]
/usr/bin/asd() [0x678379]
/lib64/libpthread.so.0() [0x38472079d1]
/lib64/libc.so.6(clone+0x6d) [0x3846ee88fd]

It is not unusual for an Aerospike server to have more than 100 threads and so it may be more readable to redirect output to a file and then use pattern matching to search for the relevant thread ID. If the problematic thread ID is found in sites.txt the sub-system that thread relates to can be found using thread-traces.

Conclusion

In general, once a memory investigation has progressed to the point where jemalloc debug data has been requested and been provided, further help from Aerospike’s core developer team would be required. This article has shown, at a basic level, how to interpret the output from jemalloc debug information, which can help in terms of increasing understanding, spotting obvious issues, and also, understanding feedback from core developers.

Notes

  • Useful Knowledge Base article discussing options for investigating memory usage within Aerospike.
  • Another useful tool in tracking down memory issues is asparsemem. This lists out memory usage by sub-system, largest first. An example output is shown below.
/work/source/modules/common/src/main/citrusleaf/cf_queue.c:136|0x686b9f|2837248
/work/source/as/src/base/scan.c:334|0x4d4ac2|9830400
/work/source/as/src/storage/drv_ssd.c:823|0x561a31|10485760
/work/source/as/src/storage/drv_ssd.c:1050|0x569ad4|83886080
/work/source/modules/ee/as/src/storage/flat_ee.c:79|0x55f806|209715200
/work/source/as/src/fabric/fabric.c:1271|0x532cdf|281804800
/work/source/as/src/fabric/fabric.c:1271 (discriminator 3)|0x5326a2|283115520
/work/source/modules/ee/as/src/base/index_ee.c:84|0x45d7c6|298188800
/work/source/as/src/storage/drv_ssd.c:293|0x560ea3|2753822720
/work/source/modules/ee/as/src/base/proto_ee.c:90|0x4634c4|32904904704
/work/source/modules/ee/as/src/storage/flat_ee.c:441|0x55fe21|33142865920

Keywords

JEMALLOC MEMORY ALLOCATION SITES.TXT JEM.TXT JEMSTATS

Timestamp

June 2021

© 2021 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.