I am running Aerospike Server Community Edition build 3.5.9. I am running in 2 node cluster using mesh networking.
I have encountered following segmentation fault →
Sep 18 2015 08:12:35 GMT: WARNING (as): (signal.c::160) SIGSEGV received, aborting Aerospike Community Edition build 3.5.9
Sep 18 2015 08:12:35 GMT: WARNING (as): (signal.c::160) SIGSEGV received, aborting Aerospike Community Edition build 3.5.9
Sep 18 2015 08:12:35 GMT: WARNING (as): (signal.c::160) SIGSEGV received, aborting Aerospike Community Edition build 3.5.9
Sep 18 2015 08:12:35 GMT: WARNING (as): (signal.c::160) SIGSEGV received, aborting Aerospike Community Edition build 3.5.9
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 0: /usr/bin/asd(as_sig_handle_segv+0x54) [0x46cf57]
Sep 18 2015 08:12:36 GMT: WARNING (udf): (udf_timer.c::72) UDF Timed Out [347288691:347288657]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 1: /lib/x86_64-linux-gnu/libc.so.6(+0x36ff0) [0x7f2558409ff0]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 2: /lib/x86_64-linux-gnu/libc.so.6(+0x98e39) [0x7f255846be39]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 3: /usr/bin/asd() [0x51d7d6]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 4: /usr/bin/asd() [0x4baee4]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 5: /usr/bin/asd(udf_aerospike__apply_update_atomic+0x2dc) [0x4bb7eb]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 6: /usr/bin/asd(udf_aerospike__execute_updates+0x2c) [0x4bbec9]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 7: /usr/bin/asd() [0x45d5a7]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 8: /usr/bin/asd() [0x517747]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 9: /usr/bin/asd() [0x53c2a8]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 0: /usr/bin/asd(as_sig_handle_segv+0x54) [0x46cf57]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 10: /usr/bin/asd(lua_pcall+0x30) [0x52bf70]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 11: /usr/bin/asd() [0x51555b]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 1: /lib/x86_64-linux-gnu/libc.so.6(+0x36ff0) [0x7f2558409ff0]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 12: /usr/bin/asd() [0x51617b]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 2: /lib/x86_64-linux-gnu/libc.so.6(+0x98e39) [0x7f255846be39]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 13: /usr/bin/asd(udf_apply_record+0x110) [0x4c28fc]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 3: /usr/bin/asd() [0x51d7d6]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 14: /usr/bin/asd(udf_rw_local+0x197) [0x4c2b69]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 4: /usr/bin/asd() [0x4baee4]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 15: /usr/bin/asd() [0x4ac998]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 16: /usr/bin/asd(as_rw_start+0x29b) [0x4ae5cd]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 5: /usr/bin/asd(udf_aerospike__apply_update_atomic+0x2dc) [0x4bb7eb]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 17: /usr/bin/asd(process_transaction+0xf48) [0x4b9332]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 6: /usr/bin/asd(udf_aerospike__execute_updates+0x2c) [0x4bbec9]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 18: /usr/bin/asd(thr_tsvc_process_or_enqueue+0x3e) [0x4b9802]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 7: /usr/bin/asd() [0x45d5a7]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 19: /usr/bin/asd(thr_demarshal+0x386) [0x481649]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f25592e2182]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 8: /usr/bin/asd() [0x517747]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 21: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f25584ce30d]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 9: /usr/bin/asd() [0x53c2a8]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 10: /usr/bin/asd(lua_pcall+0x30) [0x52bf70]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 11: /usr/bin/asd() [0x51555b]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 12: /usr/bin/asd() [0x51617b]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 13: /usr/bin/asd(udf_apply_record+0x110) [0x4c28fc]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 14: /usr/bin/asd(udf_rw_local+0x197) [0x4c2b69]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 15: /usr/bin/asd() [0x4ac998]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 16: /usr/bin/asd(as_rw_start+0x29b) [0x4ae5cd]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 17: /usr/bin/asd(process_transaction+0xf48) [0x4b9332]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 18: /usr/bin/asd(thr_tsvc_process_or_enqueue+0x3e) [0x4b9802]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 19: /usr/bin/asd(thr_demarshal+0x386) [0x481649]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f25592e2182]
Sep 18 2015 08:12:36 GMT: WARNING (as): (signal.c::162) stacktrace: frame 21: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f25584ce30d]
Both the servers crashed with the same problem. On one of the cluster node I am additionally seeing following log →
Sep 18 2015 08:14:40 GMT: WARNING (udf): (udf_aerospike.c::436) udf_aerospike_setbin: Internal Error [map-list: serialization failure (1)]... Fail
There are lot of DB writes, reads and record udf executions in my setup.