SISSEGV error causing downtime of application

1

We are using community edition aerospike server. the server however goes down giving an error SIGSEGV received, aborting Aerospike Community Edition build 5.7.0.9 os el7.

Has anyone else faced this, any help will be much appreciated.

2

Hi @abhinav_sinha welcome to the community!

Could you provide some more info about the error you’re seeing? What are you doing when you see this? What client are you using? Are you working with any Lua UDFs when this occurs?

A look at the logs from when this occurs would also be helpful. Thanks!

3

5.7.0.24 is the latest in 5.7 line, see release notes . From 5.7.0.9 up, in the release notes look at Bug Fixes section through 5.7.0.24 and see if any of those affect you. Suggest upgrade to 5.7.0.24 if you want to stay with 5.7.

4

We are using aerospike for querying hotel content and using C# client for the same. We aren’t using LUA UDFs or anything. We have formed an aerospike cluster with two servers, both the servers went down at the exact same time with following error

Nov 30 2022 00:22:20 GMT: WARNING (query): (query.c:1442) basic query job failed msg field processing

Nov 30 2022 00:22:21 GMT: WARNING (as): (signal.c:192) SIGSEGV received, aborting Aerospike Community Edition build 6.1.0.3 os el7

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:605) stacktrace: registers: rax 0000000000000001 rbx 0000000000000001 rcx 0000000039000000 rdx 00007fe5c642205f rsi 00007fe5d35e8720 rdi 007fe5c64448e000 rbp 00007fe5c641c050 rsp 00007fe5d35e8510 r8 0000000000000001 r9 000000003f800000 r10 0000000039800000 r11 0000000000000000 r12 00007fe5d35e8720 r13 00000000008f7550 r14 00007fe5d35e86f0 r15 00000000008f7540 rip 0000000000623eb3

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:618) stacktrace: found 19 frames: 0x555691 0x4acf14 0x7fe66bd688e0 0x623eb3 0x632e68 0x6335c9 0x63371b 0x633925 0x633986 0x508102 0x508bb7 0x509228 0x50d202 0x4c916e 0x4a99d0 0x5449ec 0x5448d5 0x7fe66bd5e44b 0x7fe66a90956f offset 0x0

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 0: /usr/bin/asd(cf_log_stack_trace+0xd4) [0x555691]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 1: /usr/bin/asd(as_sig_handle_segv+0x58) [0x4acf14]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 2: /lib64/libpthread.so.0(+0x118e0) [0x7fe66bd688e0]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 3: /usr/bin/asd(_ZNK9S2Polygon12MayIntersectERK6S2Cell+0xc3) [0x623eb3]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 4: /usr/bin/asd(_ZN15S2RegionCoverer12NewCandidateERK6S2Cell+0x18) [0x632e68]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 5: /usr/bin/asd(_ZN15S2RegionCoverer20GetInitialCandidatesEv+0x1a9) [0x6335c9]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 6: /usr/bin/asd(_ZN15S2RegionCoverer19GetCoveringInternalERK8S2Region+0x1b) [0x63371b]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 7: /usr/bin/asd(_ZN15S2RegionCoverer12GetCellUnionERK8S2RegionP11S2CellUnion+0x15) [0x633925]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 8: /usr/bin/asd(_ZN15S2RegionCoverer11GetCoveringERK8S2RegionPSt6vectorI8S2CellIdSaIS4_EE+0x46) [0x633986]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 9: /usr/bin/asd(geo_region_cover+0xa4) [0x508102]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 10: /usr/bin/asd() [0x508bb7]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 11: /usr/bin/asd() [0x509228]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 12: /usr/bin/asd(as_query+0x651) [0x50d202]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 13: /usr/bin/asd(as_tsvc_process_transaction+0x4ce) [0x4c916e]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 14: /usr/bin/asd() [0x4a99d0]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 15: /usr/bin/asd() [0x5449ec]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 16: /usr/bin/asd() [0x5448d5]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 17: /lib64/libpthread.so.0(+0x744b) [0x7fe66bd5e44b]

Nov 30 2022 00:22:21 GMT: WARNING (as): (log.c:628) stacktrace: frame 18: /lib64/libc.so.6(clone+0x3f) [0x7fe66a90956f]

5

What kind of query are you executing? GeoJSON? Bin has region with secondary index on it, and finding if it contains a point? or vice versa?

6

Hi Abhinav, We are currently looking into the issue and would like your assistance to help us debug further. Can you provide us with the client code that does the query which led to the crash? Also, more importantly the GEOJSON string used in the query.

Thanks, Varun

Adding addr2line for future reference

addr2line -e /usr/bin/asd -ifC 0x555691 0x4acf14 0x7fe66bd688e0 0x623eb3 0x632e68 0x6335c9 0x63371b 0x633925 0x633986 0x508102 0x508bb7 0x509228 0x50d202 0x4c916e 0x4a99d0 0x5449ec 0x5448d5 0x7fe66bd5e44b 0x7fe66a90956f
cf_log_stack_trace
/work/source/cf/src/log.c:610
reraise_signal
/work/source/as/src/base/signal.c:94
as_sig_handle_segv
/work/source/as/src/base/signal.c:195
??
??:0
S2Polygon::MayIntersect(S2Cell const&) const
/work/source/modules/s2-geometry-library/geometry/s2polygon.cc:466
S2RegionCoverer::NewCandidate(S2Cell const&)
/work/source/modules/s2-geometry-library/geometry/s2regioncoverer.cc:97
S2RegionCoverer::GetInitialCandidates()
/work/source/modules/s2-geometry-library/geometry/s2regioncoverer.cc:223 (discriminator 1)
S2RegionCoverer::GetCoveringInternal(S2Region const&)
/work/source/modules/s2-geometry-library/geometry/s2regioncoverer.cc:254
S2RegionCoverer::GetCellUnion(S2Region const&, S2CellUnion*)
/work/source/modules/s2-geometry-library/geometry/s2regioncoverer.cc:313
S2RegionCoverer::GetCovering(S2Region const&, std::vector<S2CellId, std::allocator<S2CellId> >*)
/work/source/modules/s2-geometry-library/geometry/s2regioncoverer.cc:299
geo_region_cover
/work/source/as/src/geospatial/geospatial.cc:127
range_from_msg_geojson
/work/source/as/src/query/query.c:836
get_query_range
/work/source/as/src/query/query.c:536
basic_query_job_start
/work/source/as/src/query/query.c:1439
as_query
/work/source/as/src/query/query.c:219
as_tsvc_process_transaction
/work/source/as/src/base/thr_tsvc.c:181
start_transaction
/work/source/as/src/base/service.c:912
run_service
/work/source/as/src/base/service.c:699
run_pool
/work/source/cf/src/cf_thread.c:336 (discriminator 1)
pool_shim_fn
/work/source/cf/src/cf_thread.c:378
??
??:0
??
??:0
7

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.