Single bin data-in-memory crash [Released] [Resolved]

1

I am running aerospike on two nodes and given 65G RAM to aerospike from each node. During insertion at about 7,500,00,000 records the aerospike crashed second time with SEGV. The total ram consumption at this time was 20G from each node. First one node failed and after few seconds the second node failed…

The log files from node 1


Jul 07 2014 09:46:27 GMT: INFO (info): (thr_info.c::4312)    partitions: actual 12288 sync 0 desync 0 zombie 0 wait 0 absent 0
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::69) histogram dump: reads (0 total)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::69) histogram dump: writes_master (349110803 total)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::84)  (00: 0348834224)  (01: 0000200217)  (02: 0000021609)  (03: 0000032958)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::92)  (04: 0000021773)  (05: 0000000022)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::69) histogram dump: proxy (0 total)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::69) histogram dump: writes_reply (349110804 total)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::84)  (00: 0348835034)  (01: 0000199414)  (02: 0000021606)  (03: 0000032955)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::92)  (04: 0000021773)  (05: 0000000022)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::69) histogram dump: udf (0 total)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::69) histogram dump: query (0 total)
Jul 07 2014 09:46:27 GMT: INFO (info): (hist.c::69) histogram dump: query_rec_count (0 total)
Jul 07 2014 09:46:27 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata3 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:2 ms total-time:2 ms
Jul 07 2014 09:46:27 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 09:46:28 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata2 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4228)  system memory: free 73826648kb ( 74 percent free )
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4235)  migrates in progress ( 0 , 0 ) ::: ClusterSize 1 ::: objects 350254138
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4243)  rec refs 350338172 ::: rec locks 0 ::: trees 0 ::: wr reqs 1 ::: mig tx 0 ::: mig rx 0
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4249)  replica errs :: null 0 non-null 0 ::: sync copy errs :: node 0 :: master 0
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4257)    trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 0 ::: dq 0 : fds - proto (47, 5865, 5818) : hb 0 : fab 16
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4259)    heartbeat_received: self 23916 : foreign 46733
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4272)    tree_counts: nsup 1 scan 0 batch 0 dup 0 wprocess 0 migrx 0 migtx 0 ssdr 0 ssdw 0 rw 3
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4288) namespace test: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4288) namespace salman: disk inuse: 44833964032 memory inuse: 22416266048 (bytes) sindex memory inuse: 0 (bytes) avail pct 81
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4288) namespace bar: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 09:46:37 GMT: INFO (info): (thr_info.c::4312)    partitions: actual 12288 sync 0 desync 0 zombie 0 wait 0 absent 0
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::69) histogram dump: reads (0 total)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::69) histogram dump: writes_master (350254491 total)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::84)  (00: 0349977721)  (01: 0000200228)  (02: 0000021630)  (03: 0000033021)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::92)  (04: 0000021870)  (05: 0000000022)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::69) histogram dump: proxy (0 total)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::69) histogram dump: writes_reply (350254494 total)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::84)  (00: 0349978532)  (01: 0000199425)  (02: 0000021627)  (03: 0000033018)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::92)  (04: 0000021870)  (05: 0000000022)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::69) histogram dump: udf (0 total)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::69) histogram dump: query (0 total)
Jul 07 2014 09:46:37 GMT: INFO (info): (hist.c::69) histogram dump: query_rec_count (0 total)
Jul 07 2014 09:46:38 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata3 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 09:46:38 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:2 ms total-time:2 ms
Jul 07 2014 09:46:39 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata2 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:2 ms total-time:2 ms
Jul 07 2014 09:46:41 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /ssd-data/aerospikedata3: free 65624M contig 65624M  w-q 0 w-free 62584 swb-free 1 w-tot 14214
Jul 07 2014 09:46:41 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /ssd-data/aerospikedata: free 65446M contig 65446M  w-q 0 w-free 62415 swb-free 1 w-tot 14383
Jul 07 2014 09:46:41 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /ssd-data/aerospikedata2: free 65623M contig 65623M  w-q 0 w-free 62583 swb-free 1 w-tot 14215
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4228)  system memory: free 73755120kb ( 74 percent free )
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4235)  migrates in progress ( 0 , 0 ) ::: ClusterSize 1 ::: objects 351369613
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4243)  rec refs 351414955 ::: rec locks 1 ::: trees 0 ::: wr reqs 3 ::: mig tx 0 ::: mig rx 0
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4249)  replica errs :: null 0 non-null 0 ::: sync copy errs :: node 0 :: master 0
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4257)    trans_in_progress: wr 4 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 0 ::: dq 0 : fds - proto (47, 5883, 5836) : hb 0 : fab 16
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4259)    heartbeat_received: self 23982 : foreign 46733
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4272)    tree_counts: nsup 1 scan 0 batch 0 dup 0 wprocess 0 migrx 0 migtx 0 ssdr 0 ssdw 0 rw 3
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4288) namespace test: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4288) namespace salman: disk inuse: 44976570368 memory inuse: 22487657024 (bytes) sindex memory inuse: 0 (bytes) avail pct 81
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4288) namespace bar: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 09:46:47 GMT: INFO (info): (thr_info.c::4312)    partitions: actual 12288 sync 0 desync 0 zombie 0 wait 0 absent 0
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::69) histogram dump: reads (0 total)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::69) histogram dump: writes_master (351370013 total)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::84)  (00: 0351093015)  (01: 0000200248)  (02: 0000021665)  (03: 0000033082)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::92)  (04: 0000021984)  (05: 0000000022)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::69) histogram dump: proxy (0 total)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::69) histogram dump: writes_reply (351370018 total)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::84)  (00: 0351093826)  (01: 0000199445)  (02: 0000021662)  (03: 0000033079)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::92)  (04: 0000021984)  (05: 0000000022)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::69) histogram dump: udf (0 total)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::69) histogram dump: query (0 total)
Jul 07 2014 09:46:47 GMT: INFO (info): (hist.c::69) histogram dump: query_rec_count (0 total)
Jul 07 2014 09:46:49 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata3 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 09:46:49 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 09:46:50 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata2 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:2 ms total-time:2 ms
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::148) Signal SEGV received: stack trace
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 0: /usr/bin/asd(as_sig_handle_segv+0x43) [0x464951]
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 1: /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fc6662af4a0]
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 2: /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4) [0x7fc666f61e84]
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 3: /usr/bin/asd() [0x53bded]
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 4: /usr/bin/asd(cf_free_at+0x7e) [0x4e1d78]
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 5: /usr/bin/asd() [0x454f4a]
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 6: /usr/bin/asd() [0x488db3]
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 7: /usr/bin/asd(thr_nsup+0x137f) [0x48a7df]
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fc666f5fe9a]
Jul 07 2014 09:46:55 GMT: WARNING (as): (signal.c::155) stacktrace: frame 9: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fc66636d3fd]

Log from second node :

Jul 07 2014 10:04:07 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:2 ms total-time:2 ms
Jul 07 2014 10:04:07 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata2 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:2 ms total-time:2 ms
Jul 07 2014 10:04:07 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata3 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4228)  system memory: free 76751176kb ( 77 percent free )
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4235)  migrates in progress ( 0 , 0 ) ::: ClusterSize 1 ::: objects 335519744
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4243)  rec refs 335631609 ::: rec locks 0 ::: trees 0 ::: wr reqs 0 ::: mig tx 0 ::: mig rx 0
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4249)  replica errs :: null 0 non-null 0 ::: sync copy errs :: node 0 :: master 0
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4257)    trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 0 ::: dq 0 : fds - proto (0, 1472, 1472) : hb 0 : fab 16
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4259)    heartbeat_received: self 4350 : foreign 0
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4272)    tree_counts: nsup 1 scan 0 batch 0 dup 0 wprocess 0 migrx 0 migtx 0 ssdr 0 ssdw 0 rw 0
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4288) namespace test: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4288) namespace salman: disk inuse: 42946527232 memory inuse: 21473263616 (bytes) sindex memory inuse: 0 (bytes) avail pct 82
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4288) namespace bar: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 10:04:13 GMT: INFO (info): (thr_info.c::4312)    partitions: actual 12288 sync 0 desync 0 zombie 0 wait 0 absent 0
Jul 07 2014 10:04:13 GMT: INFO (info): (hist.c::69) histogram dump: reads (0 total)
Jul 07 2014 10:04:13 GMT: INFO (info): (hist.c::69) histogram dump: writes_master (0 total)
Jul 07 2014 10:04:13 GMT: INFO (info): (hist.c::69) histogram dump: proxy (0 total)
Jul 07 2014 10:04:13 GMT: INFO (info): (hist.c::69) histogram dump: writes_reply (0 total)
Jul 07 2014 10:04:13 GMT: INFO (info): (hist.c::69) histogram dump: udf (0 total)
Jul 07 2014 10:04:13 GMT: INFO (info): (hist.c::69) histogram dump: query (0 total)
Jul 07 2014 10:04:13 GMT: INFO (info): (hist.c::69) histogram dump: query_rec_count (0 total)
Jul 07 2014 10:04:18 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /ssd-data/aerospikedata3: free 66270M contig 66270M  w-q 0 w-free 63200 swb-free 0 w-tot 0
Jul 07 2014 10:04:18 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /ssd-data/aerospikedata2: free 66270M contig 66270M  w-q 0 w-free 63200 swb-free 0 w-tot 0
Jul 07 2014 10:04:18 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /ssd-data/aerospikedata: free 66102M contig 66102M  w-q 0 w-free 63040 swb-free 0 w-tot 0
Jul 07 2014 10:04:18 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 10:04:18 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata2 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:2 ms total-time:2 ms
Jul 07 2014 10:04:18 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata3 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4228)  system memory: free 76753772kb ( 77 percent free )
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4235)  migrates in progress ( 0 , 0 ) ::: ClusterSize 1 ::: objects 335519744
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4243)  rec refs 335540213 ::: rec locks 0 ::: trees 0 ::: wr reqs 0 ::: mig tx 0 ::: mig rx 0
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4249)  replica errs :: null 0 non-null 0 ::: sync copy errs :: node 0 :: master 0
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4257)    trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 0 ::: dq 0 : fds - proto (0, 1495, 1495) : hb 0 : fab 16
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4259)    heartbeat_received: self 4416 : foreign 0
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4272)    tree_counts: nsup 1 scan 0 batch 0 dup 0 wprocess 0 migrx 0 migtx 0 ssdr 0 ssdw 0 rw 0
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4288) namespace test: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4288) namespace salman: disk inuse: 42946527232 memory inuse: 21473263616 (bytes) sindex memory inuse: 0 (bytes) avail pct 82
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4288) namespace bar: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 10:04:23 GMT: INFO (info): (thr_info.c::4312)    partitions: actual 12288 sync 0 desync 0 zombie 0 wait 0 absent 0
Jul 07 2014 10:04:23 GMT: INFO (info): (hist.c::69) histogram dump: reads (0 total)
Jul 07 2014 10:04:23 GMT: INFO (info): (hist.c::69) histogram dump: writes_master (0 total)
Jul 07 2014 10:04:23 GMT: INFO (info): (hist.c::69) histogram dump: proxy (0 total)
Jul 07 2014 10:04:23 GMT: INFO (info): (hist.c::69) histogram dump: writes_reply (0 total)
Jul 07 2014 10:04:23 GMT: INFO (info): (hist.c::69) histogram dump: udf (0 total)
Jul 07 2014 10:04:23 GMT: INFO (info): (hist.c::69) histogram dump: query (0 total)
Jul 07 2014 10:04:23 GMT: INFO (info): (hist.c::69) histogram dump: query_rec_count (0 total)
Jul 07 2014 10:04:29 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:2 ms total-time:2 ms
Jul 07 2014 10:04:29 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata2 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 10:04:29 GMT: INFO (drv_ssd): (drv_ssd.c::950) /ssd-data/aerospikedata3 defrag curr_pos 76799 wblocks:0 recs:0 waits:0 lock-time:1 ms total-time:1 ms
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4228)  system memory: free 76751444kb ( 77 percent free )
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4235)  migrates in progress ( 0 , 0 ) ::: ClusterSize 1 ::: objects 335519744
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4243)  rec refs 335677178 ::: rec locks 0 ::: trees 0 ::: wr reqs 0 ::: mig tx 0 ::: mig rx 0
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4249)  replica errs :: null 0 non-null 0 ::: sync copy errs :: node 0 :: master 0
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4257)    trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 0 ::: dq 0 : fds - proto (0, 1518, 1518) : hb 0 : fab 16
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4259)    heartbeat_received: self 4483 : foreign 0
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4272)    tree_counts: nsup 1 scan 0 batch 0 dup 0 wprocess 0 migrx 0 migtx 0 ssdr 0 ssdw 0 rw 0
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4288) namespace test: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4288) namespace salman: disk inuse: 42946527232 memory inuse: 21473263616 (bytes) sindex memory inuse: 0 (bytes) avail pct 82
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4288) namespace bar: disk inuse: 0 memory inuse: 0 (bytes) sindex memory inuse: 0 (bytes) avail pct 100
Jul 07 2014 10:04:33 GMT: INFO (info): (thr_info.c::4312)    partitions: actual 12288 sync 0 desync 0 zombie 0 wait 0 absent 0
Jul 07 2014 10:04:33 GMT: INFO (info): (hist.c::69) histogram dump: reads (0 total)
Jul 07 2014 10:04:33 GMT: INFO (info): (hist.c::69) histogram dump: writes_master (0 total)
Jul 07 2014 10:04:33 GMT: INFO (info): (hist.c::69) histogram dump: proxy (0 total)
Jul 07 2014 10:04:33 GMT: INFO (info): (hist.c::69) histogram dump: writes_reply (0 total)
Jul 07 2014 10:04:33 GMT: INFO (info): (hist.c::69) histogram dump: udf (0 total)
Jul 07 2014 10:04:33 GMT: INFO (info): (hist.c::69) histogram dump: query (0 total)
Jul 07 2014 10:04:33 GMT: INFO (info): (hist.c::69) histogram dump: query_rec_count (0 total)
Jul 07 2014 10:04:38 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /ssd-data/aerospikedata3: free 66270M contig 66270M  w-q 0 w-free 63200 swb-free 0 w-tot 0
Jul 07 2014 10:04:38 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /ssd-data/aerospikedata2: free 66270M contig 66270M  w-q 0 w-free 63200 swb-free 0 w-tot 0
Jul 07 2014 10:04:38 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /ssd-data/aerospikedata: free 66102M contig 66102M  w-q 0 w-free 63040 swb-free 0 w-tot 0
2

Would it be possible to to get the version of Aerospike server and client that you are using and also the operating system being used?

3

The aerospike server i am using is aerospike-server-community-3.3.5-ubuntu12.04 and using java client

<groupId>com.aerospike</groupId>
    <artifactId>aerospike-client</artifactId>
    <version>3.0.26</version>
  </dependency>
4

My machine is installed with 96GB RAM on each node.

Kernel Linux plat4 3.11.0-15-generic #25~precise1-Ubuntu SMP Thu Jan 30 17:39:31 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

OS Distributor ID: Ubuntu Description: Ubuntu 12.04.4 LTS Release: 12.04 Codename: precise

5

I’ve opened a ticket internally for our engineers to investigate. Would it be possible for you to provide your test code in order for us to replicate this case.

6

I have a huge file with about 1760 million records. They are in form of CSV. 3343,231,3434

According my logic i have to take first two integers as key and third field as value so here is my code

public static void IngestSData (String inputFileName ) throws IOException, AerospikeException
   {
      
      Host[] hosts = new Host[] {
             new Host("172.20.48.153", 3000), new Host("172.20.48.155",3000)
            
         };
      
      AerospikeClient client = new AerospikeClient(new ClientPolicy(), hosts);
      
      WritePolicy policy = new WritePolicy();
      policy.timeout = 5000 ;
      
      
      BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(inputFileName)));
      String line = br.readLine();
      
      //line = br.readLine();
      Key k ;
      Bin b ;
      long counter = 0 ;
      int i = 1 ;
      ByteBuffer key = ByteBuffer.allocate(8);
      while (line != null )
      {
         
         String [] tokens = line.split(",");
         key.putInt(Integer.parseInt(tokens[0]));
         key.putInt(Integer.parseInt(tokens[1]));
         key.rewind();
         k = new Key ("salman" ,"n", key.array() );
         b = new Bin ("s",Integer.parseInt(tokens[2]));
         client.add(policy, k,b);
         
         key.clear();
         line = br.readLine() ;
      }
      client.close();
      br.close();
   }
7

One more thing that it is crashed only with SINGLE-BIN settled to be true.

8

Thanks. We are working on a fix that should resolve this issue with single bin in-memory. A work around would be to try and disable data-in-memory. Please try your test with data-in-memory false.

9

Issue has been resolved with latest release.