The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
Startup fails with memory error when Aerospike runs under systemd
Problem Description
When running under systemd the Aerospike start up fails and the following messages are displayed in the log:
Oct 14 2019 09:41:01 GMT: FAILED ASSERTION (namespace): (namespace_ee.c:550) can't remove persistent memory base block: Operation not permitted
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:210) SIGUSR1 received, aborting Aerospike Enterprise Edition build 4.5.3.6 os debian9
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: registers: rax 0000000000000000 rbx 000000000000000a rcx 00007f89fc106fbf rdx 0000000000000000 rsi 00007ffc983a1bb0 rdi 0000000000000002 rbp 00007ffc983a21e0 rsp 00007ffc983a1c28 r8 0000000000000000 r9 00007ffc983a1bb0 r10 0000000000000008 r11 0000000000000246 r12 000000000000001f r13 0000000000000000 r14 000000000000008f r15 000056493758e8c0 rip 00007f89fc106fbf
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: found 10 frames: 0x56493517332c 0x7f89fc1070e0 0x7f89fc106fbf 0x56493538288e 0x564935101e61 0x56493510335f 0x5649351452e5 0x5649350fafb2 0x7f89faeb42e1 0x5649350fb9ba offset 0x56493507d000
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 0: /usr/bin/asd(as_sig_handle_usr1+0x11f) [0x56493517332c]
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x7f89fc1070e0]
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 2: /lib/x86_64-linux-gnu/libpthread.so.0(raise+0xcf) [0x7f89fc106fbf]
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 3: /usr/bin/asd(cf_fault_event+0x231) [0x56493538288e]
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 4: /usr/bin/asd(xmem_must_delete_base_block+0x63) [0x564935101e61]
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 5: /usr/bin/asd(as_namespaces_setup+0x93b) [0x56493510335f]
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 6: /usr/bin/asd(as_namespaces_init+0x14) [0x5649351452e5]
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 7: /usr/bin/asd(main+0x302) [0x5649350fafb2]
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 8: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f89faeb42e1]
Oct 14 2019 09:41:01 GMT: WARNING (as): (signal.c:212) stacktrace: frame 9: /usr/bin/asd(_start+0x2a) [0x5649350fb9ba]
Explanation
This error will occur when Aerospike has been started by a different user and the shared memory segments used by the primary index are still allocated. Specifically the stack trace occurs when shared memory has been allocated previously by a user and the user used to startup Aerospike does not have privileges to access those shared memory segments.
Solution
In Aerospike version 4.5.3.2 and newer, the user will be read from the aerospike.conf file however, if the shared memory segments are still allocated by the previous user it may be necessary to force a cold start. If this does not work, Enterprise Licensees can contact Aerospike support for advice on how to release shared memory segments.
Notes
- Refer to the Running as Non Root documentation for further details.
- Relevant KB article: Changing user and group of asd under systemd.
Keywords
XMEM_MUST_DELETE_BASE_BLOCK XMEM MUST DELETE BASE BLOCK PERSISTENT MEMORY BASE BLOCK NOT PERMITTED
Timestamp
October 2019