Aerospike Python client Segmentation Fault

Mangat_Modi · March 30, 2016, 7:00am

I am reading a key from aerospike, modify value and write it again:-

aerospike_dca_client.put(key,bin)
...
...
bls_key_a = (AEROSPIKE_NAMESPACE,SET_NAME,bls_key)
hp_bid_str_a = aerospike_dca_client.get(bls_key_a)[3];

Now the issue is that put is working good, but during I get a segmentation fault. I did debug using gdb and got the stacktrace:

Program received signal SIGSEGV, Segmentation fault.
as_shm_node_get (cluster=0x353bd90508, ns=0x7fffffffc851 "test", digest=0x7fffffffc8e1 "/\231!\221h\223\240\021KX\377\357O\346u\214q\356\267#\177", write=false, replica=AS_POLICY_REPLICA_MASTER)
    at src/main/aerospike/as_shm_cluster.c:431

> 431 src/main/aerospike/as_shm_cluster.c: No such file or directory.

        in src/main/aerospike/as_shm_cluster.c
        Missing separate debuginfos, use: debuginfo-install blas-3.2.1-4.el6.x86_64
        (gdb) py-bt
        Undefined command: "py-bt".  Try "help".
        (gdb) bt
#0  as_shm_node_get (cluster=0x353bd90508, ns=0x7fffffffc851 "test", digest=0x7fffffffc8e1 "/\231!\221h\223\240\021KX\377\357O\346u\214q\356\267#\177", write=false, replica=AS_POLICY_REPLICA_MASTER)
            at src/main/aerospike/as_shm_cluster.c:431
#1  0x00007fffdf219ce5 in as_node_get (cluster=0x353bd90508, err=0x7fffffffc920, cn=0x7fffffffc7c0, command=0x7fffffffc740 "\002\003", command_len=82, timeout_ms=1000, retry=1,
                parse_results_fn=0x7fffdf21a97b <as_command_parse_result>, parse_results_data=0x7fffffffc7e0) at src/include/aerospike/as_cluster.h:562
#2  as_command_execute (cluster=0x353bd90508, err=0x7fffffffc920, cn=0x7fffffffc7c0, command=0x7fffffffc740 "\002\003", command_len=82, timeout_ms=1000, retry=1,
                    parse_results_fn=0x7fffdf21a97b <as_command_parse_result>, parse_results_data=0x7fffffffc7e0) at src/main/aerospike/as_command.c:435
#3  0x00007fffdf211a67 in aerospike_key_get (as=0x17dead0, err=0x7fffffffc920, policy=0x17dfbb0, key=<value optimized out>, rec=0x7fffffffc848) at src/main/aerospike/aerospike_key.c:114
#4  0x00007fffdf1e4295 in AerospikeClient_Get_Invoke (self=0x720d30, py_key=0x7fffb0b05730, py_policy=<value optimized out>) at src/main/client/get.c:96
#5  0x00007fffdf1e44a0 in AerospikeClient_Get (self=0x720d30, args=<value optimized out>, kwds=<value optimized out>) at src/main/client/get.c:174
#6  0x00007ffff7d12f24 in call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4033
#7  PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#8  0x00007ffff7d1399e in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4119
#9  call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#10 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#11 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7ffff7b0ee30, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=1, kws=0x7ffff7bc6068, kwcount=0,
                        defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3265
#12 0x00007ffff7c927c8 in function_call (func=0x7fffdf1936e0, arg=0x7fffb0af9450, kw=0x7fffde2dfd70) at Objects/funcobject.c:526
#13 0x00007ffff7c631a3 in PyObject_Call (func=0x7fffdf1936e0, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:2529
#14 0x00007ffff7d11746 in ext_do_call (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4346
#15 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2718
#16 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7fffe041f6b0, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=1, kws=0x1e142b0, kwcount=0, defs=0x0,
                            defcount=0, closure=0x7fffde39bbe0) at Python/ceval.c:3265
#17 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#18 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#19 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#20 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7fffe042b1b0, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=4, kws=0x7fffcbc517e0, kwcount=0,
                                defs=0x7fffdffa8428, defcount=1, closure=0x0) at Python/ceval.c:3265
#21 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#22 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#23 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#24 0x00007ffff7d1399e in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4119
#25 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#26 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#27 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7fffe041f730, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=2, kws=0x1d6c958, kwcount=0, defs=0x0,
                                    defcount=0, closure=0x0) at Python/ceval.c:3265
#28 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#29 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#30 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#31 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7ffff1ac6030, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=1, kws=0x7fffde5b85b8, kwcount=1,
                                        defs=0x7fffdf18bfa8, defcount=1, closure=0x0) at Python/ceval.c:3265
#32 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#33 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#34 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#35 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7ffff1ac6130, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=0, kws=0x662d48, kwcount=1,
                                            defs=0x7fffdf18bfe8, defcount=1, closure=0x0) at Python/ceval.c:3265
#36 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#37 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#38 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#39 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7ffff1ac6230, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0,
                                                defcount=0, closure=0x0) at Python/ceval.c:3265
                                                ---Type <return> to continue, or q <return> to quit---
#40 0x00007ffff7d14bb2 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>) at Python/ceval.c:673
#41 0x00007ffff7d34850 in run_mod (fp=0x65f7b0, filename=<value optimized out>, start=<value optimized out>, globals=0x7ffff7b9b168, locals=0x7ffff7b9b168, closeit=1, flags=0x7fffffffe0e0)
                                                    at Python/pythonrun.c:1377
#42 PyRun_FileExFlags (fp=0x65f7b0, filename=<value optimized out>, start=<value optimized out>, globals=0x7ffff7b9b168, locals=0x7ffff7b9b168, closeit=1, flags=0x7fffffffe0e0) at Python/pythonrun.c:1363
#43 0x00007ffff7d34a2f in PyRun_SimpleFileExFlags (fp=0x65f7b0, filename=0x7fffffffe4d9 "bidlandscape_sf.py", closeit=1, flags=0x7fffffffe0e0) at Python/pythonrun.c:955
#44 0x00007ffff7d4a194 in Py_Main (argc=<value optimized out>, argv=<value optimized out>) at Modules/main.c:640
#45 0x000000353ba1ed5d in __libc_start_main (main=0x400710 <main>, argc=7, ubp_av=0x7fffffffe208, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>,
                                                        stack_end=0x7fffffffe1f8) at libc-start.c:226
#46 0x0000000000400649 in _start ()

It looks like aerospike_key_get wants to access some subroutine in file src/main/aerospike/as_shm_cluster.c which it is unable to find hence segmentation fault. I checked my $LD_LIBRARY_PATH and it does contain directory containing aerospike.so.

Why this particular file is missing? Is it somewhere else other than aerospike.so?
Why segmentation fault. It should be gracefully handled in Python client

rbotzer · March 30, 2016, 7:54am

Please open this as an issue with the aerospike/aerospike-client-python repo on Github. It would be good if you could provide a script that reproduces the problem for you, rather than a couple of lines.

Please state the version of Python you’re using, the client release numer that you’re using, the OS. Also, whether you built it from source, or installed it with pip. Thanks.

Mangat_Modi · March 30, 2016, 11:08am

@rbotzer

I am finally able to reproduce the bug. Issue is with connecting aerospike client again without closing it first. Somehow I believed that Aerospike client uses connection pool, so we could make multiple connections and it will just reuse the connections. But no, it fails with the segmentation fault instead of an exception. Core dump is same as above for the code below:-

import aerospike
import traceback
#sys.settrace
AEROSPIKE_NAMESPACE = "test";
BIDLAND_SET = "bid_landscape";
aerospike_dca = {'hosts':[('127.0.0.1',3000)]}
aerospike_dca_client = aerospike.client(aerospike_dca);
for i in xrange(1,3):
 aerospike_dca_client.connect()
 try:
  bls_key_a = (AEROSPIKE_NAMESPACE,SET_NAME, "hello")
  hp_bid_str_a = aerospike_dca_client.get(bls_key_a)[2];
  print str(hp_bid_str_a)
 except:
  traceback.print_exc()
 bin = {'bidland_string':"world"}
 key = (AEROSPIKE_NAMESPACE,SET_NAME,"hello")
 aerospike_dca_client.put(key,bin)
 print str(aerospike_dca_client.get(bls_key_a))

following is my configuration:-

Host OS: Linux, Centos 6
Aerospike Version: Aerospike Community Edition build 3.7.0.2 (installed using pip)
Python: 2.7.2

rbotzer · March 30, 2016, 1:41pm

Thanks, that helps. For now, please close and open explicitly to work around the issue.

Mangat_Modi · March 30, 2016, 3:56pm

Yup, did that. Thanks for your help!

Topic		Replies	Views
Segmentation Fault Node.js Client	2	2438	May 10, 2016
Seg fault on python client udf	1	1630	January 11, 2016
Segmentation fault in client only with more than one node in server C Client Library	4	2190	August 7, 2015
Aerospike C Client Crash - SIGSEGV C Client Library	2	1120	March 12, 2018
Segmentation fault (core dumped) when adding GeoJSON polygon in Aerospike Client Libraries index	2	1549	May 22, 2017

Aerospike Python client Segmentation Fault

Related topics