Aeropike crashing during scan job execution

maxulan · December 14, 2014, 9:06pm

Env: machine: Amazon EC2 m3.large OS: Ubuntu 12.04.4 LTS Server: Aerospike Community Edition build 3.3.19 Client: AQL - aerospike-tools Version: 3.3.18 Configuration: single node, /etc/aerospike/aerospike.conf is standard

query which caused crash:

aql> select sid, id, r_m from feed.products 
2014-12-15 01:06:54 ERROR network error: errno 9 fd 4
Error: (-1) AEROSPIKE_ERR_CLIENT

Log:

Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:509) Adding bin |r_m| to projected bins |(null)| 
Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:617)  Binname projected inside is |id| 
Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:617)  Binname projected inside is |r_m| 
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_send_response_to_client:1131) tid 129: response to client fd 71 bytes 862
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 24, scanned partitions= 38, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 0 objects from partition 27
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 27, scanned partitions= 39, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 1 objects from partition 30
Dec 14 2014 12:06:54 GMT: DEBUG (index): (index.c:as_index_reduce_partial:928) as_index_reduce_traverse took 0 ms
Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:502)  Binname projected inside is |id| 
Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:509) Adding bin |id| to projected bins |(null)| 
Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:502)  Binname projected inside is |r_m| 
Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:509) Adding bin |r_m| to projected bins |(null)| 
Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:617)  Binname projected inside is |id| 
Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:617)  Binname projected inside is |r_m| 
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_send_response_to_client:1131) tid 129: response to client fd 71 bytes 1042
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 30, scanned partitions= 40, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 0 objects from partition 33
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 33, scanned partitions= 41, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 0 objects from partition 36
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 36, scanned partitions= 42, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 1 objects from partition 39
Dec 14 2014 12:06:54 GMT: DEBUG (index): (index.c:as_index_reduce_partial:928) as_index_reduce_traverse took 0 ms
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 39, scanned partitions= 43, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 2 objects from partition 42
Dec 14 2014 12:06:54 GMT: DEBUG (index): (index.c:as_index_reduce_partial:928) as_index_reduce_traverse took 0 ms
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 42, scanned partitions= 44, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 0 objects from partition 45
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 45, scanned partitions= 45, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 0 objects from partition 48
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 48, scanned partitions= 46, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 0 objects from partition 51
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 51, scanned partitions= 47, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 0 objects from partition 54
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 54, scanned partitions= 48, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 0 objects from partition 57
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 57, scanned partitions= 49, job id=129
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 0 objects from partition 60
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 0: /usr/bin/asd(as_sig_handle_segv+0x54) [0x466828]
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1684) Workitem id= 60, scanned partitions= 50, job id=129
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 1: /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f26da4704a0]
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1731) finished workitem
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 2: /lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x29f1) [0x7f26da4853b1]
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1529) waiting for workitem
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 3: /lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0xb0) [0x7f26da5434c0]
Dec 14 2014 12:06:54 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1595) scan_partition: need to scan 1 objects from partition 63
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 4: /usr/bin/asd(cf_fault_event+0x1fb) [0x4ec3b2]
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 5: /usr/bin/asd(as_msg_make_response_bufbuilder+0x595) [0x45ff39]
Dec 14 2014 12:06:54 GMT: DEBUG (index): (index.c:as_index_reduce_partial:928) as_index_reduce_traverse took 0 ms
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 6: /usr/bin/asd(tscan_tree_reduce+0x405) [0x4a9069]
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 7: /usr/bin/asd(as_index_reduce_partial+0x186) [0x456e1a]
Dec 14 2014 12:06:54 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:502)  Binname projected inside is |id| 
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 8: /usr/bin/asd(tscan_partition_thr+0x2c3) [0x4aa49c]
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f26db120e9a]
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::157) stacktrace: frame 10: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f26da52e3fd]
Dec 14 2014 12:06:54 GMT: WARNING (as): (signal.c::150) SIGSEGV received, aborting Aerospike Community Edition build 3.3.19

maxulan · December 14, 2014, 11:51pm

More info and update on the bug:

In select id, r_m from test.products query: column id is a secondary index and r_m might be null.
Issue is reproducable on the latest version of Aerospike: Aerospike Community Edition build 3.4.0

    Dec 14 2014 23:36:03 GMT: INFO (info): (thr_info.c::4594) namespace feed: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 129280 (bytes) avail pct 100
    Dec 14 2014 23:36:03 GMT: INFO (info): (thr_info.c::4594) namespace test: disk inuse: 0 memory inuse: 750522 (bytes) sindex memory inuse: 221450 (bytes) avail pct 100
    Dec 14 2014 23:36:03 GMT: DEBUG (partition): (partition.c:as_partition_getstates:793) partitions: migrating 0, rejecting writes 0.
    Dec 14 2014 23:36:03 GMT: INFO (info): (thr_info.c::4619)    partitions: actual 12288 sync 0 desync 0 zombie 0 wait 0 absent 0
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::137) histogram dump: reads (1456 total) msec
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::163)  (00: 0000001456)
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::137) histogram dump: writes_master (1638 total) msec
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::163)  (00: 0000001638)
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::137) histogram dump: proxy (0 total) msec
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::137) histogram dump: writes_reply (1638 total) msec
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::163)  (00: 0000001638)
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::137) histogram dump: udf (3350 total) msec
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::163)  (00: 0000003349) (01: 0000000001)
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::137) histogram dump: query (1 total) msec
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::163)  (03: 0000000001)
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::137) histogram dump: query_rec_count (1 total) count
    Dec 14 2014 23:36:03 GMT: INFO (info): (hist.c::163)  (09: 0000000001)
    Dec 14 2014 23:36:03 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16227 changed from arena #9 ==> #9
    Dec 14 2014 23:36:03 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:627) [Sending Info request via fast path.]
    Dec 14 2014 23:36:03 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal_reaper_fn:188) reaper: mismatched connection count: 11 in stats vs 10 calculated
    Dec 14 2014 23:36:04 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16224 changed from arena #7 ==> #7
    Dec 14 2014 23:36:04 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:627) [Sending Info request via fast path.]
    Dec 14 2014 23:36:04 GMT: DEBUG (sindex): (secondary_index.c:as_sindex_reserve:965) Index test_images_idx_adv_id in 2 state Reserved to reference count 2 < 2 at base/thr
    _sindex.c:261
    Dec 14 2014 23:36:04 GMT: DEBUG (sindex): (secondary_index.c:as_sindex_release:1014) Index test_images_idx_adv_id in 2 state Released to reference count 1 < 2 at base/th
    r_sindex.c:442
    Dec 14 2014 23:36:04 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16228 changed from arena #10 ==> #10
    Dec 14 2014 23:36:04 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:627) [Sending Info request via fast path.]
    Dec 14 2014 23:36:04 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16226 changed from arena #8 ==> #8
    Dec 14 2014 23:36:04 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:627) [Sending Info request via fast path.]
    Dec 14 2014 23:36:04 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16224 changed from arena #7 ==> #7
    Dec 14 2014 23:36:04 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:627) [Sending Info request via fast path.]
    Dec 14 2014 23:36:04 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16226 changed from arena #8 ==> #8
    Dec 14 2014 23:36:04 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:627) [Sending Info request via fast path.]
    Dec 14 2014 23:36:04 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16227 changed from arena #9 ==> #9
    Dec 14 2014 23:36:04 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:627) [Sending Info request via fast path.]
    Dec 14 2014 23:36:04 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16224 changed from arena #7 ==> #7
    Dec 14 2014 23:36:04 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:627) [Sending Info request via fast path.]
    Dec 14 2014 23:36:04 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16224 changed from arena #7 ==> #7
    Dec 14 2014 23:36:04 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:627) [Sending Info request via fast path.]
    Dec 14 2014 23:36:04 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal_reaper_fn:188) reaper: mismatched connection count: 11 in stats vs 10 calculated
    Dec 14 2014 23:36:05 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 16228 changed from arena #10 ==> #152
    Dec 14 2014 23:36:05 GMT: DEBUG (tsvc): (thr_tsvc.c:process_transaction:463) Received Scan Request: TrID(0)
    Dec 14 2014 23:36:05 GMT: INFO (scan): (thr_tscan.c::743) scan job received
    Dec 14 2014 23:36:05 GMT: INFO (scan): (thr_tscan.c::794) scan_option 0x8 0x64
    Dec 14 2014 23:36:05 GMT: INFO (scan): (thr_tscan.c::879) scan option: Fail if cluster change True
    Dec 14 2014 23:36:05 GMT: INFO (scan): (thr_tscan.c::880) scan option: Background Job False
    Dec 14 2014 23:36:05 GMT: INFO (scan): (thr_tscan.c::881) scan option: priority is 0 n_threads 3 job_type 1
    Dec 14 2014 23:36:05 GMT: INFO (scan): (thr_tscan.c::882) scan option: scan_pct is 100 
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 1 objects from partition 0
    Dec 14 2014 23:36:05 GMT: DEBUG (index): (index.c:as_index_reduce_partial:926) as_index_reduce_traverse took 0 ms
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:502)  Binname projected inside is |sid| 
    
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:509) Adding bin |sid| to projected bins |(null)| 
    
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:502)  Binname projected inside is |id| 
    
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:509) Adding bin |id| to projected bins |(null)| 
    
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:502)  Binname projected inside is |r_m| 
    
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:506) To be projected bin |r_m| not found 
    
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:617)  Binname projected inside is |sid| 
    
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:617)  Binname projected inside is |id| 
    
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:617)  Binname projected inside is |r_m| 
    
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 2
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_send_response_to_client:1208) tid 1: response to client fd 65 bytes 105
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 1, scanned partitions= 1, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 2, scanned partitions= 2, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 4
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 5
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 4, scanned partitions= 3, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 5, scanned partitions= 4, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 7
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 8
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 7, scanned partitions= 5, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 144
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 144, scanned partitions= 177, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 1 objects from partition 147
    Dec 14 2014 23:36:05 GMT: DEBUG (index): (index.c:as_index_reduce_partial:926) as_index_reduce_traverse took 0 ms
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 147, scanned partitions= 178, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 150
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 150, scanned partitions= 179, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 1 objects from partition 153
    Dec 14 2014 23:36:05 GMT: DEBUG (index): (index.c:as_index_reduce_partial:926) as_index_reduce_traverse took 0 ms
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 153, scanned partitions= 180, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 156
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 156, scanned partitions= 181, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 159
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 159, scanned partitions= 182, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 1 objects from partition 162
    Dec 14 2014 23:36:05 GMT: DEBUG (index): (index.c:as_index_reduce_partial:926) as_index_reduce_traverse took 0 ms
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:502)  Binname projected inside is |sid| 
    
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::150) SIGSEGV received, aborting Aerospike Community Edition build 3.4.0
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 0: /usr/bin/asd(as_sig_handle_segv+0x54) [0x467838]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 1: /lib/x86_64-linux-gnu/libc.so.6(+0x36150) [0x7f0959a4b150]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 2: /lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x29f1) [0x7f0959a60061]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 3: /lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0xb0) [0x7f0959b1e3e0]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 4: /usr/bin/asd(cf_fault_event+0x1fb) [0x4edcd2]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 5: /usr/bin/asd(as_msg_make_response_bufbuilder+0x595) [0x460fa6]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 6: /usr/bin/asd(tscan_tree_reduce+0x401) [0x4aa0c9]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 7: /usr/bin/asd(as_index_reduce_partial+0x186) [0x457c83]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 8: /usr/bin/asd(tscan_partition_thr+0x2ca) [0x4aae32]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f095a910e9a]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 10: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f0959b0931d]
    
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 144
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 144, scanned partitions= 177, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 1 objects from partition 147
    Dec 14 2014 23:36:05 GMT: DEBUG (index): (index.c:as_index_reduce_partial:926) as_index_reduce_traverse took 0 ms
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 147, scanned partitions= 178, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 150
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 150, scanned partitions= 179, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 1 objects from partition 153
    Dec 14 2014 23:36:05 GMT: DEBUG (index): (index.c:as_index_reduce_partial:926) as_index_reduce_traverse took 0 ms
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 153, scanned partitions= 180, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 156
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 156, scanned partitions= 181, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 0 objects from partition 159
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1713) Workitem id= 159, scanned partitions= 182, job id=1
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1763) finished workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1558) waiting for workitem
    Dec 14 2014 23:36:05 GMT: DEBUG (scan): (thr_tscan.c:tscan_partition_thr:1624) scan_partition: need to scan 1 objects from partition 162
    Dec 14 2014 23:36:05 GMT: DEBUG (index): (index.c:as_index_reduce_partial:926) as_index_reduce_traverse took 0 ms
    Dec 14 2014 23:36:05 GMT: DEBUG (proto): (proto.c:as_msg_make_response_bufbuilder:502)  Binname projected inside is |sid| 
    
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::150) SIGSEGV received, aborting Aerospike Community Edition build 3.4.0
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 0: /usr/bin/asd(as_sig_handle_segv+0x54) [0x467838]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 1: /lib/x86_64-linux-gnu/libc.so.6(+0x36150) [0x7f0959a4b150]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 2: /lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x29f1) [0x7f0959a60061]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 3: /lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0xb0) [0x7f0959b1e3e0]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 4: /usr/bin/asd(cf_fault_event+0x1fb) [0x4edcd2]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 5: /usr/bin/asd(as_msg_make_response_bufbuilder+0x595) [0x460fa6]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 6: /usr/bin/asd(tscan_tree_reduce+0x401) [0x4aa0c9]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 7: /usr/bin/asd(as_index_reduce_partial+0x186) [0x457c83]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 8: /usr/bin/asd(tscan_partition_thr+0x2ca) [0x4aae32]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f095a910e9a]
    Dec 14 2014 23:36:05 GMT: WARNING (as): (signal.c::157) stacktrace: frame 10: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f0959b0931d]

maxulan · December 15, 2014, 1:55am

Issue resolved after replacing unicast network configuration with mesh mode.

Topic		Replies	Views
Aerospike Community Edition Crashing (AER-3780) [Released] [Resolved] Node.js Client	8	2952	August 1, 2015
5.0.0.4 crashes after scanning system linux	5	792	June 18, 2020
Aerospike cluster crashed after index creation Configuration aws	5	2841	September 22, 2015
Scans suddenly stop responding Query & Indexing	4	1965	February 9, 2016
Scan bug identified (CLIENT-1637) Support Bulletins	7	2155	October 10, 2022

Aeropike crashing during scan job execution

Related topics