New node fails to start with SIGFPE received


#1

New node fails to start with SIGFPE received

Problem Description

A newly created node will not start. In the logs the following is observed.

Jun 15 2016 15:28:17 GMT: INFO (nsup): (thr_nsup.c::1289) namespace supervisor started
Jun 15 2016 15:28:17 GMT: INFO (demarshal): (thr_demarshal.c::255) Saved original JEMalloc arena #5 for thr_demarshal()
Jun 15 2016 15:28:17 GMT: INFO (demarshal): (thr_demarshal.c::283) Service started: socket 3000
Jun 15 2016 15:28:18 GMT: INFO (demarshal): (thr_demarshal.c::255) Saved original JEMalloc arena #6 for thr_demarshal()
Jun 15 2016 15:28:18 GMT: INFO (demarshal): (thr_demarshal.c::255) Saved original JEMalloc arena #7 for thr_demarshal()
Jun 15 2016 15:28:18 GMT: INFO (demarshal): (thr_demarshal.c::255) Saved original JEMalloc arena #8 for thr_demarshal()
Jun 15 2016 15:28:19 GMT: INFO (demarshal): (thr_demarshal.c::792) Waiting to spawn demarshal threads ...
Jun 15 2016 15:28:19 GMT: INFO (demarshal): (thr_demarshal.c::795) Started 4 Demarshal Threads
Jun 15 2016 15:28:19 GMT: INFO (as): (as.c::450) service ready: soon there will be cake!
Jun 15 2016 15:28:22 GMT: WARNING (as): (signal.c::104) SIGFPE received, aborting Aerospike Enterprise Edition build 3.5.14
Jun 15 2016 15:28:22 GMT: WARNING (as): (signal.c::106) stacktrace: found 5 frames
Jun 15 2016 15:28:22 GMT: WARNING (as): (signal.c::106) stacktrace: frame 0: /usr/bin/asd(as_sig_handle_fpe+0x54) [0x47e959]
Jun 15 2016 15:28:22 GMT: WARNING (as): (signal.c::106) stacktrace: frame 1: /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f34df81cd40]
Jun 15 2016 15:28:22 GMT: WARNING (as): (signal.c::106) stacktrace: frame 2: /usr/bin/asd(as_sindex__defrag_fn+0x1a8) [0x4c4270]
Jun 15 2016 15:28:22 GMT: WARNING (as): (signal.c::106) stacktrace: frame 3: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f34e06f4182]
Jun 15 2016 15:28:22 GMT: WARNING (as): (signal.c::106) stacktrace: frame 4: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f34df8e047d]

Subsequent retries result in the same behaviour.

Explanation

This error will occur when there are no namespaces defined in aerospike.conf. The function as_sindex__defrag_fn is the first function which tries to access the namespace array to do garbage collection, this happens whether there are secondary indexes configured or not.

Solution

Complete the namespace stanza in /opt/aerospike/aerospike.conf with a test namespace. If there are running nodes in the cluster the namespace definitions must match for the node in question to join the cluster.

Notes

  • Information on the namespace stanza can be found here:

http://www.aerospike.com/docs/operations/configure/namespace

Keywords

NAMESPACE SIGFPE

Timestamp

6/16/16.