1422:LDT-Sub Record Open Error received


#1

hi, The error which is next to the suddenness in LDT processing occurred.

WARNING (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::2563) [ERROR]<ldt_common_2014_12_20.A:openSubRec()> SubRec Open Failure: Digest(EE C6 49 C4 B0 76 A7 15 BC 7F 06 17 9A EC 00 00 00 00 00 00)

A similar error reply occurs by data addition processing after error occurrence, and it can’t be added. Is there a way to recover this problem?


#2

Can you indicate which version of Aerospike was used, and code snippet that can reproduce this error? Are you able to the scan the LDT record afterwards?

Another diagnostic tool is to run, using “aql”: execute llist.dump(’’) on .[] where pk =

and post the diagnostic result, which will be in the aerospike.log, starting with the following:

::::: P R I N T T R E E :::::


#3

thank you for your reply

  1. I use aerospike version 3.5.9

    ldt topRec key is used url format such like “http://hoge.com/2000/1/hvacstat

    ldt subRec key is used timestamp format .

  2. sorry, when error occured , I have no log what data was inserted.

  3. query status is

-llist.size => I can get result

-llist.find_fist => I can get result

-llist.find_last => I can get result

-llist.scan => error occur

result of query is below.

aql> execute llist.size(‘foo’) on <namespace_name>.<sets_name> where pk=‘http://hoge.com/G/2000/30/6/PointOfName

±------+

| size |

±------+

| 25519 |

±------+

1 row in set (0.000 secs)

aql> execute llist.find_first(‘foo’,1) on <namespace_name>.<sets_name> where pk=‘http://hoge.com/G/2000/30/6/PointOfName

±----------------------------------------------------------------------------------------------------------------+ | find_first | ±----------------------------------------------------------------------------------------------------------------+ | [{“description”:"", “time”:“2015-02-14 13:42:28.228+09”, “value”:“0”, “timestamp”:1423888948, “key”:1423888948}] | ±----------------------------------------------------------------------------------------------------------------+

aql> execute llist.find_last(‘foo’,1) on <namespace_name>.<sets_name> where pk=‘http://hoge.com/G/2000/30/6/PointOfName

±---------------------------------------------------------------------------------------------------------------+ | find_last | ±---------------------------------------------------------------------------------------------------------------+ | [{“description”:"", “time”:“2015-04-23 13:58:00+09”, “value”:“65.0”, “timestamp”:1429765080, “key”:1429765080}] | ±---------------------------------------------------------------------------------------------------------------+

aql> execute llist.scan(‘foo’) on <namespace_name>.<sets_name> where pk='http://hoge.com/G/2000/30/6/PointOfName’ Error: (100) /opt/aerospike/sys/udf/lua/ldt/ldt_common.lua:751: 1422:LDT-Sub Record Open Error

aql> execute llist.scan(‘foo’,1429745180,1429785180) on <namespace_name>.<sets_name> where pk='http://hoge.com/G/2000/30/6/PointOfName’ Error: (100) /opt/aerospike/sys/udf/lua/ldt/ldt_common.lua:751: 1422:LDT-Sub Record Open Error

  1. llist.dump() is now executing.but it takes so long time. when it is completed, I post them later.

#5

sorry for wating. diagnostic result is below.

May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::6678) <PT> <PT> <PT> :::::   P R I N T   T R E E  ::::: <PT>
<PT>
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1772) ROOT::Bin(pointvalue)
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1772) ROOT::Keys: Size(1) List([1425193380])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1772) ROOT: size (0), CompactList ([])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1772) ROOT:: Left Leaf 45 02 98 DF 7E 0A CB 15 99 A4 B8 EE 1E
 BC 00 00 00 00 00 00 Right Leaf 45 02 98 DF 7E 0A CB 16 99 5E 6B 0F FD 00 00 00 00 00 00 00
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1790) NODE::Level(2)
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1790) NODE::Digest(45 02 98 DF 7E 0A CB 15 99 79 40 89 41 F9
00 00 00 00 00 00)
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1790) NODE::KeyList: Size(201) List([1424226360, 1424228040,
1424230440, 1424232120, 1424233260, 1424234040, 1424235060, 1424235720, 1424237820, 1424239740, 1424244840, 1424247060, 1424250120, 1424255160, 14
24257680, 1424261520, 1424263500, 1424265660, 1424268300, 1424271720, 1424273520, 1424276640, 1424278800, 1424280720, 1424283600, 1424287140, 1424
289300, 1424291520, 1424293320, 1424295480, 1424295840, 1424296980, 1424299680, 1424304420, 1424306520, 1424309880, 1424311860, 1424313180, 142431
4260, 1424315460, 1424318100, 1424319300, 1424320080, 1424320440, 1424321520, 1424322780, 1424324040, 1424325720, 1424329320, 1424332920, 14243351
40, 1424335620, 1424336280, 1424337060, 1424337960, 1424339220, 1424340720, 1424342820, 1424397000, 1424402220, 1424407860, 1424410080, 1424412660
, 1424416500, 1424421420, 1424426640, 1424431860, 1424437080, 1424442300, 1424447460, 1424452680, 1424457900, 1424463120, 1424468340, 1424473560,
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1790) NODE::Level(2)
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1790) NODE::Digest(45 02 98 DF 7E 0A CB 15 99 95 40 89 41 F9
00 00 00 00 00 00)
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1790) NODE::KeyList: Size(203) List([1425198600, 1425203820,
1425209040, 1425214260, 1425219480, 1425224700, 1425229920, 1425235140, 1425240360, 1425245580, 1425250800, 1425256080, 1425261300, 1425266520, 14
25271680, 1425276900, 1425282120, 1425287340, 1425292560, 1425297780, 1425303000, 1425308220, 1425313440, 1425318660, 1425323820, 1425329040, 1425
334260, 1425339480, 1425344640, 1425349860, 1425355080, 1425360300, 1425365520, 1425370740, 1425375960, 1425381240, 1425386460, 1425391680, 142539
6900, 1425402180, 1425407400, 1425412560, 1425414600, 1425416280, 1425417060, 1425417360, 1425419460, 1425421920, 1425424020, 1425426120, 14254282
20, 1425430320, 1425432420, 1425434580, 1425436680, 1425438780, 1425440880, 1425442920, 1425445020, 1425447120, 1425449220, 1425451080, 1425452880
, 1425454740, 1425456600, 1425458460, 1425460200, 1425462060, 1425463920, 1425465780, 1425467640, 1425469500, 1425471360, 1425473160, 1425474900,
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(37) Size(3073) Key([1423888948, 14
24012402, 1424224200, 1424224260, 1424224320, 1424224380, 1424224440, 1424224500, 1424224560, 1424224620, 1424224680, 1424224740, 1424224860, 1424
224920, 1424224980, 1424225040, 1424225100, 1424225160, 1424225220, 1424225280, 1424225340, 1424225400, 1424225460, 1424225520, 1424225580, 142422
5640, 1424225700, 1424225760, 1424225820, 1424225880, 1424225940, 1424226000, 1424226060, 1424226120, 1424226180, 1424226240, 1424226300])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(26) Size(2162) Key([1424226360, 14
24226420, 1424226480, 1424226540, 1424226600, 1424226780, 1424226840, 1424226900, 1424226960, 1424227020, 1424227080, 1424227140, 1424227200, 1424
227260, 1424227320, 1424227380, 1424227440, 1424227500, 1424227560, 1424227620, 1424227680, 1424227740, 1424227800, 1424227860, 1424227920, 142422
7980])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(39) Size(3244) Key([1424228040, 14
24228100, 1424228160, 1424228220, 1424228280, 1424228340, 1424228460, 1424228520, 1424228580, 1424228640, 1424228700, 1424228760, 1424228820, 1424
228880, 1424228940, 1424229000, 1424229060, 1424229120, 1424229180, 1424229240, 1424229300, 1424229360, 1424229420, 1424229480, 1424229540, 142422
9600, 1424229660, 1424229720, 1424229780, 1424229840, 1424229900, 1424229960, 1424230020, 1424230080, 1424230140, 1424230200, 1424230260, 14242303
20, 1424230380])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(28) Size(2355) Key([1424230440, 14
24230500, 1424230560, 1424230620, 1424230680, 1424230740, 1424230800, 1424230860, 1424230920, 1424230980, 1424231040, 1424231100, 1424231160, 1424
231220, 1424231280, 1424231340, 1424231400, 1424231460, 1424231520, 1424231580, 1424231640, 1424231700, 1424231760, 1424231820, 1424231880, 142423
1940, 1424232000, 1424232060])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(19) Size(1599) Key([1424232120, 14
24232180, 1424232240, 1424232300, 1424232360, 1424232420, 1424232480, 1424232540, 1424232600, 1424232660, 1424232720, 1424232780, 1424232840, 1424
232900, 1424232960, 1424233020, 1424233080, 1424233140, 1424233200])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(13) Size(1093) Key([1424233260, 14
24233320, 1424233380, 1424233440, 1424233500, 1424233560, 1424233620, 1424233680, 1424233740, 1424233800, 1424233860, 1424233920, 1424233980])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(17) Size(1431) Key([1424234040, 14
24234100, 1424234160, 1424234220, 1424234280, 1424234340, 1424234400, 1424234460, 1424234520, 1424234580, 1424234640, 1424234700, 1424234760, 1424
234820, 1424234880, 1424234940, 1424235000])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(11) Size(925) Key([1424235060, 142
4235120, 1424235180, 1424235240, 1424235300, 1424235360, 1424235420, 1424235480, 1424235540, 1424235600, 1424235660])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(35) Size(2943) Key([1424235720, 14
24235780, 1424235840, 1424235900, 1424235960, 1424236020, 1424236080, 1424236140, 1424236200, 1424236260, 1424236320, 1424236380, 1424236440, 1424
236500, 1424236560, 1424236620, 1424236680, 1424236740, 1424236800, 1424236860, 1424236920, 1424236980, 1424237040, 1424237100, 1424237160, 142423
7220, 1424237280, 1424237340, 1424237400, 1424237460, 1424237520, 1424237580, 1424237640, 1424237700, 1424237760])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(32) Size(2691) Key([1424237820, 14
24237880, 1424237940, 1424238000, 1424238060, 1424238120, 1424238180, 1424238240, 1424238300, 1424238360, 1424238420, 1424238480, 1424238540, 1424
238600, 1424238660, 1424238720, 1424238780, 1424238840, 1424238900, 1424238960, 1424239020, 1424239080, 1424239140, 1424239200, 1424239260, 142423
9320, 1424239380, 1424239440, 1424239500, 1424239560, 1424239620, 1424239680])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(85) Size(7143) Key([1424239740, 14
24239800, 1424239860, 1424239920, 1424239980, 1424240040, 1424240100, 1424240160, 1424240220, 1424240280, 1424240340, 1424240400, 1424240460, 1424
240520, 1424240580, 1424240640, 1424240700, 1424240760, 1424240820, 1424240880, 1424240940, 1424241000, 1424241060, 1424241120, 1424241180, 142424
1240, 1424241300, 1424241360, 1424241420, 1424241480, 1424241540, 1424241600, 1424241660, 1424241720, 1424241780, 1424241840, 1424241900, 14242419
60, 1424242020, 1424242080, 1424242140, 1424242200, 1424242260, 1424242320, 1424242380, 1424242440, 1424242500, 1424242560, 1424242620, 1424242680
, 1424242740, 1424242800, 1424242860, 1424242920, 1424242980, 1424243040, 1424243100, 1424243160, 1424243220, 1424243280, 1424243340, 1424243400,
1424243460, 1424243520, 1424243580, 1424243640, 1424243700, 1424243760, 1424243820, 1424243880, 1424243940, 1424244000, 1424244060, 1424244120, 1
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(37) Size(3111) Key([1424244840, 14
24244900, 1424244960, 1424245020, 1424245080, 1424245140, 1424245200, 1424245260, 1424245320, 1424245380, 1424245440, 1424245500, 1424245560, 1424
245620, 1424245680, 1424245740, 1424245800, 1424245860, 1424245920, 1424245980, 1424246040, 1424246100, 1424246160, 1424246220, 1424246280, 142424
6340, 1424246400, 1424246460, 1424246520, 1424246580, 1424246640, 1424246700, 1424246760, 1424246820, 1424246880, 1424246940, 1424247000])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(51) Size(4287) Key([1424247060, 14
24247120, 1424247180, 1424247240, 1424247300, 1424247360, 1424247420, 1424247480, 1424247540, 1424247600, 1424247660, 1424247720, 1424247780, 1424
247840, 1424247900, 1424247960, 1424248020, 1424248080, 1424248140, 1424248200, 1424248260, 1424248320, 1424248380, 1424248440, 1424248500, 142424
8560, 1424248620, 1424248680, 1424248740, 1424248800, 1424248860, 1424248920, 1424248980, 1424249040, 1424249100, 1424249160, 1424249220, 14242492
80, 1424249340, 1424249400, 1424249460, 1424249520, 1424249580, 1424249640, 1424249700, 1424249760, 1424249820, 1424249880, 1424249940, 1424250000
, 1424250060])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(84) Size(7059) Key([1424250120, 14
24250180, 1424250240, 1424250300, 1424250360, 1424250420, 1424250480, 1424250540, 1424250600, 1424250660, 1424250720, 1424250780, 1424250840, 1424
250900, 1424250960, 1424251020, 1424251080, 1424251140, 1424251200, 1424251260, 1424251320, 1424251380, 1424251440, 1424251500, 1424251560, 142425
1620, 1424251680, 1424251740, 1424251800, 1424251860, 1424251920, 1424251980, 1424252040, 1424252100, 1424252160, 1424252220, 1424252280, 14242523
40, 1424252400, 1424252460, 1424252520, 1424252580, 1424252640, 1424252700, 1424252760, 1424252820, 1424252880, 1424252940, 1424253000, 1424253060
, 1424253120, 1424253180, 1424253240, 1424253300, 1424253360, 1424253420, 1424253480, 1424253540, 1424253600, 1424253660, 1424253720, 1424253780,
1424253840, 1424253900, 1424253960, 1424254020, 1424254080, 1424254140, 1424254200, 1424254260, 1424254320, 1424254380, 1424254440, 1424254500, 1
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(42) Size(3531) Key([1424255160, 14
24255220, 1424255280, 1424255340, 1424255400, 1424255460, 1424255520, 1424255580, 1424255640, 1424255700, 1424255760, 1424255820, 1424255880, 1424
255940, 1424256000, 1424256060, 1424256120, 1424256180, 1424256240, 1424256300, 1424256360, 1424256420, 1424256480, 1424256540, 1424256600, 142425
6660, 1424256720, 1424256780, 1424256840, 1424256900, 1424256960, 1424257020, 1424257080, 1424257140, 1424257200, 1424257260, 1424257320, 14242573
80, 1424257440, 1424257500, 1424257560, 1424257620])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(64) Size(5379) Key([1424257680, 14
24257740, 1424257800, 1424257860, 1424257920, 1424257980, 1424258040, 1424258100, 1424258160, 1424258220, 1424258280, 1424258340, 1424258400, 1424
258460, 1424258520, 1424258580, 1424258640, 1424258700, 1424258760, 1424258820, 1424258880, 1424258940, 1424259000, 1424259060, 1424259120, 142425
9180, 1424259240, 1424259300, 1424259360, 1424259420, 1424259480, 1424259540, 1424259600, 1424259660, 1424259720, 1424259780, 1424259840, 14242599
00, 1424259960, 1424260020, 1424260080, 1424260140, 1424260200, 1424260260, 1424260320, 1424260380, 1424260440, 1424260500, 1424260560, 1424260620
, 1424260680, 1424260740, 1424260800, 1424260860, 1424260920, 1424260980, 1424261040, 1424261100, 1424261160, 1424261220, 1424261280, 1424261340,
1424261400, 1424261460])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(33) Size(2775) Key([1424261520, 14
24261580, 1424261640, 1424261700, 1424261760, 1424261820, 1424261880, 1424261940, 1424262000, 1424262060, 1424262120, 1424262180, 1424262240, 1424
262300, 1424262360, 1424262420, 1424262480, 1424262540, 1424262600, 1424262660, 1424262720, 1424262780, 1424262840, 1424262900, 1424262960, 142426
3020, 1424263080, 1424263140, 1424263200, 1424263260, 1424263320, 1424263380, 1424263440])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(36) Size(3027) Key([1424263500, 14
24263560, 1424263620, 1424263680, 1424263740, 1424263800, 1424263860, 1424263920, 1424263980, 1424264040, 1424264100, 1424264160, 1424264220, 1424
264280, 1424264340, 1424264400, 1424264460, 1424264520, 1424264580, 1424264640, 1424264700, 1424264760, 1424264820, 1424264880, 1424264940, 142426
5000, 1424265060, 1424265120, 1424265180, 1424265240, 1424265300, 1424265360, 1424265420, 1424265480, 1424265540, 1424265600])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(44) Size(3699) Key([1424265660, 14
24265720, 1424265780, 1424265840, 1424265900, 1424265960, 1424266020, 1424266080, 1424266140, 1424266200, 1424266260, 1424266320, 1424266380, 1424
266440, 1424266500, 1424266560, 1424266620, 1424266680, 1424266740, 1424266800, 1424266860, 1424266920, 1424266980, 1424267040, 1424267100, 142426
7160, 1424267220, 1424267280, 1424267340, 1424267400, 1424267460, 1424267520, 1424267580, 1424267640, 1424267700, 1424267760, 1424267820, 14242678
80, 1424267940, 1424268000, 1424268060, 1424268120, 1424268180, 1424268240])
May 15 2015 21:24:55 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(57) Size(4791) Key([1424268300, 14
24268360, 1424268420, 1424268480, 1424268540, 1424268600, 1424268660, 1424268720, 1424268780, 1424268840, 1424268900, 1424268960, 1424269020, 1424
269080, 1424269140, 1424269200, 1424269260, 1424269320, 1424269380, 1424269440, 1424269500, 1424269560, 1424269620, 1424269680, 1424269740, 142426
9800, 1424269860, 1424269920, 1424269980, 1424270040, 1424270100, 1424270160, 1424270220, 1424270280, 1424270340, 1424270400, 1424270460, 14242705
20, 1424270580, 1424270640, 1424270700, 1424270760, 1424270820, 1424270880, 1424270940, 1424271000, 1424271060, 1424271120, 1424271180, 1424271240
, 1424271300, 1424271360, 1424271420, 1424271480, 1424271540, 1424271600, 1424271660])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(30) Size(2523) Key([1424271720, 14
24271780, 1424271840, 1424271900, 1424271960, 1424272020, 1424272080, 1424272140, 1424272200, 1424272260, 1424272320, 1424272380, 1424272440, 1424
272500, 1424272560, 1424272620, 1424272680, 1424272740, 1424272800, 1424272860, 1424272920, 1424272980, 1424273040, 1424273100, 1424273160, 142427
3220, 1424273280, 1424273340, 1424273400, 1424273460])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(52) Size(4371) Key([1424273520, 14
24273580, 1424273640, 1424273700, 1424273760, 1424273820, 1424273880, 1424273940, 1424274000, 1424274060, 1424274120, 1424274180, 1424274240, 1424
274300, 1424274360, 1424274420, 1424274480, 1424274540, 1424274600, 1424274660, 1424274720, 1424274780, 1424274840, 1424274900, 1424274960, 142427
5020, 1424275080, 1424275140, 1424275200, 1424275260, 1424275320, 1424275380, 1424275440, 1424275500, 1424275560, 1424275620, 1424275680, 14242757
40, 1424275800, 1424275860, 1424275920, 1424275980, 1424276040, 1424276100, 1424276160, 1424276220, 1424276280, 1424276340, 1424276400, 1424276460
, 1424276520, 1424276580])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(36) Size(3027) Key([1424276640, 14
24276700, 1424276760, 1424276820, 1424276880, 1424276940, 1424277000, 1424277060, 1424277120, 1424277180, 1424277240, 1424277300, 1424277360, 1424
277420, 1424277480, 1424277540, 1424277600, 1424277660, 1424277720, 1424277780, 1424277840, 1424277900, 1424277960, 1424278020, 1424278080, 142427
8140, 1424278200, 1424278260, 1424278320, 1424278380, 1424278440, 1424278500, 1424278560, 1424278620, 1424278680, 1424278740])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(32) Size(2691) Key([1424278800, 14
24278860, 1424278920, 1424278980, 1424279040, 1424279100, 1424279160, 1424279220, 1424279280, 1424279340, 1424279400, 1424279460, 1424279520, 1424
279580, 1424279640, 1424279700, 1424279760, 1424279820, 1424279880, 1424279940, 1424280000, 1424280060, 1424280120, 1424280180, 1424280240, 142428
0300, 1424280360, 1424280420, 1424280480, 1424280540, 1424280600, 1424280660])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(48) Size(4035) Key([1424280720, 14
24280780, 1424280840, 1424280900, 1424280960, 1424281020, 1424281080, 1424281140, 1424281200, 1424281260, 1424281320, 1424281380, 1424281440, 1424
281500, 1424281560, 1424281620, 1424281680, 1424281740, 1424281800, 1424281860, 1424281920, 1424281980, 1424282040, 1424282100, 1424282160, 142428
2220, 1424282280, 1424282340, 1424282400, 1424282460, 1424282520, 1424282580, 1424282640, 1424282700, 1424282760, 1424282820, 1424282880, 14242829
40, 1424283000, 1424283060, 1424283120, 1424283180, 1424283240, 1424283300, 1424283360, 1424283420, 1424283480, 1424283540])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(59) Size(4959) Key([1424283600, 14
24283660, 1424283720, 1424283780, 1424283840, 1424283900, 1424283960, 1424284020, 1424284080, 1424284140, 1424284200, 1424284260, 1424284320, 1424
284380, 1424284440, 1424284500, 1424284560, 1424284620, 1424284680, 1424284740, 1424284800, 1424284860, 1424284920, 1424284980, 1424285040, 142428
5100, 1424285160, 1424285220, 1424285280, 1424285340, 1424285400, 1424285460, 1424285520, 1424285580, 1424285640, 1424285700, 1424285760, 14242858
20, 1424285880, 1424285940, 1424286000, 1424286060, 1424286120, 1424286180, 1424286240, 1424286300, 1424286360, 1424286420, 1424286480, 1424286540
, 1424286600, 1424286660, 1424286720, 1424286780, 1424286840, 1424286900, 1424286960, 1424287020, 1424287080])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(36) Size(3027) Key([1424287140, 14
24287200, 1424287260, 1424287320, 1424287380, 1424287440, 1424287500, 1424287560, 1424287620, 1424287680, 1424287740, 1424287800, 1424287860, 1424
287920, 1424287980, 1424288040, 1424288100, 1424288160, 1424288220, 1424288280, 1424288340, 1424288400, 1424288460, 1424288520, 1424288580, 142428
8640, 1424288700, 1424288760, 1424288820, 1424288880, 1424288940, 1424289000, 1424289060, 1424289120, 1424289180, 1424289240])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(37) Size(3111) Key([1424289300, 14
24289360, 1424289420, 1424289480, 1424289540, 1424289600, 1424289660, 1424289720, 1424289780, 1424289840, 1424289900, 1424289960, 1424290020, 1424
290080, 1424290140, 1424290200, 1424290260, 1424290320, 1424290380, 1424290440, 1424290500, 1424290560, 1424290620, 1424290680, 1424290740, 142429
0800, 1424290860, 1424290920, 1424290980, 1424291040, 1424291100, 1424291160, 1424291220, 1424291280, 1424291340, 1424291400, 1424291460])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(30) Size(2523) Key([1424291520, 14
24291580, 1424291640, 1424291700, 1424291760, 1424291820, 1424291880, 1424291940, 1424292000, 1424292060, 1424292120, 1424292180, 1424292240, 1424
292300, 1424292360, 1424292420, 1424292480, 1424292540, 1424292600, 1424292660, 1424292720, 1424292780, 1424292840, 1424292900, 1424292960, 142429
3020, 1424293080, 1424293140, 1424293200, 1424293260])
May 15 2015 21:24:56 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1793) LEAF::Objects: Count(36) Size(3027) Key([1424293320, 14
24293380, 1424293440, 1424293500, 1424293560, 1424293620, 1424293680, 1424293740, 1424293800, 1424293860, 1424293920, 1424293980, 1424294040, 1424
294100, 1424294160, 1424294220, 1424294280, 1424294340, 1424294400, 1424294460, 1424294520, 1424294580, 1424294640, 1424294700, 1424294760, 142429
4820, 1424294880, 1424294940, 1424295000, 1424295060, 1424295120, 1424295180, 1424295240, 1424295300, 1424295360, 1424295420])
May 15 2015 21:24:56 GMT: INFO (ldt): (ldt_aerospike.c::750) 45 02 98 DF 7E 0A CB 15 99 ED 65 6D 37 EB 00 00 00 00 00 00 Failed to open Sub Record
 rv=-3 110365357366082<Digest>:0x450298df7e0acb1599ed656d37eb64606febdb42
May 15 2015 21:24:56 GMT: WARNING (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1782) [ERROR]<ldt_common_2014_12_20.A:openSubRec()> SubRec
 Open Failure: Digest(45 02 98 DF 7E 0A CB 15 99 ED 65 6D 37 EB 00 00 00 00 00 00)

#6

If you have range_delete and updates in your Application call. We have recently found and fixed bug, fix for this should show up in next release …

– R


#7

hi raj , thank you so much! we only call llist.update() in our Application.(but, we don’t use range_delete) so we wait for next release.

<?php $status=$this->ldt_topRec_obj->find($bin[‘timestamp’],$record); if ( $status != Aerospike::OK ) $status=$this->ldt_topRec_obj->update($bin); else $status=$this->ldt_topRec_obj->add($bin);

is this case same of what you show to me?

Thanks,


#8

Caz_nagai,

Are you doing updates with item sizes which differ considerably. e.g insert 1K item into list and update it with 10k items … or some such way. What is your write block size configuration ?

– R


#9

so thanks raj

Item do you mean the size of the one record of sub-record of the size? Or do you mean the number of records in the sub record?

our procedure is

  • First of all, you, regardless of the presence or absence of sub-record, to create a top record.
  • Then, for the top record, we will continue to put a sub-record.
  • Sub records sequentially, will be added from time to time

I think as it is not the data size of the one sub-record increases or decreases suddenly. However, that there is a big difference in the size of each record in the sub-record is assumed.

In addition, the setting of the write-block-size is not doing anything. This is the default setting. We use HDD Storage Engine with Data in Memory.

Again, I try to learn the Linux Capacity Planning Guide again.

thanks


#10

Just to be clear on terminology. LLIST is collection of items. System maintains records and sub-records to store these items. Please note that a record/sub-record can store multiple such items.

Question is what is size of item that you add to the list, i.e if the call you make is llist.add(, ). And is the size variation high for a single LLIST.

– R


#11

I’m also seeing these errors (as well as many more related to LDT and not).

In this case, I’m using llists to queue ads for individual domains on a high-volume bidding server. Items added to the list vary in size from 400 bytes to 15k, lists from 10-10000 items long. The 8-node cluster is hit 50k-80k times per second, insertions maybe 4k-5k/second.

I haven’t found a reliable way to reproduce the errors I see. I’m using llist.add(), llist.remove_all() for most manipulations.

Is there any additional insight on how to analyze what’s going on?

This is on 3.6.0-1.


#12

I face this problem in 3.5.15 with llist.range queries


#13

I just upgraded to 3.6.2 this morning. After about 60-90 minutes, I continue to get these errors:

/opt/aerospike/sys/udf/lua/ldt/ldt_common.lua:751: 1422:LDT-Sub Record Open Error

#14

Dougluce,

Did you do rolling upgrade from <3.6.*. Or was it cluster restart.

  • R

#15

It was a cluster restart. All data was dropped.


#16

@dougluce can you describe more what data you have and what operations you are making to see this happen?


#17

Hi, I am also facing similar issue on Aerospike Community Edition build 3.6.2

aql> execute llist.size(‘MINUTE’) on test.gst_36 where PK=0; ±-----+ | size | ±-----+ | 12 | ±-----+ 1 row in set (0.001 secs)

aql> execute llist.find_first(‘MINUTE’,1) on test.gst_36 where PK=0; ±---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | find_first | ±---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | [{“vo”:AC ED 00 05 73 72 00 24 63 6F 6D 2E 74 72 79 6D 61 61 72 2E 73 65 72 76 69 63 65 2E 73 74 61 74 73 2E 53 74 61 74 4F 62 6A 65 63 74 00 00 00 00 00 00 00 2A 02 00 02 49 00 03 6B 65 79 4C 00 09 76 61 6C 75 65 73 4D 61 70 74 00 0F 4C 6A 61 76 61 2F 7 | ±---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.001 secs)

aql> execute llist.find_last(‘MINUTE’,1) on test.gst_36 where PK=0; Error: (100) /opt/aerospike/sys/udf/lua/ldt/ldt_common.lua:751: 1422:LDT-Sub Record Open Error

aql> execute llist.scan(‘MINUTE’) on test.gst_36 where PK=0; Error: (100) /opt/aerospike/sys/udf/lua/ldt/ldt_common.lua:751: 1422:LDT-Sub Record Open Error

aql> execute llist.dump(‘MINUTE’) on test.gst_36 where PK=0; Error: (100) /opt/aerospike/sys/udf/lua/ldt/ldt_common.lua:751: 1422:LDT-Sub Record Open Error

Oct 22 2015 17:40:33 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::6704) ::::: P R I N T T R E E ::::: Oct 22 2015 17:40:33 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1775) ROOT::Bin(MINUTE) Oct 22 2015 17:40:33 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1775) ROOT::Keys: Size(1) List([24085930]) Oct 22 2015 17:40:33 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1775) ROOT: size (0), CompactList ([]) Oct 22 2015 17:40:33 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1775) ROOT:: Left Leaf FF 30 BA 2E 6E 00 00 22 03 11 FC 07 9D 5C 00 00 00 00 00 00 Right Leaf FF 30 BA 2E 6F 00 00 22 03 54 FC 07 9D 5C 00 00 00 00 00 00 Oct 22 2015 17:40:33 GMT: INFO (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1796) LEAF::Objects: Count(1) Size(220) Key([-1]) Oct 22 2015 17:40:33 GMT: INFO (ldt): (ldt_aerospike.c::752) FF 30 BA 2E 6F 00 00 22 03 54 FC 07 9D 5C 00 00 00 00 00 00 Failed to open Sub Record rv=-3 1873726780438:0xff30ba2e6f0000220354fc079d5c01b442d1b416 Oct 22 2015 17:40:33 GMT: WARNING (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1785) [ERROR]<ldt_common_2014_12_20.A:openSubRec()> SubRec Open Failure: Digest(FF 30 BA 2E 6F 00 00 22 03 54 FC 07 9D 5C 00 00 00 00 00 00) Parent Digest(NULL)

Is there anything that i can do to get rid of this error?


#18

I Use Aerospike Community Edition build 3.6.4

I also get these without logic or pattern.

aql> EXECUTE llist.scan(‘listBin’) on test.lists WHERE PK='K-12345’ Error: (100) /opt/aerospike/sys/udf/lua/ldt/ldt_common.lua:751: 1422:LDT-Sub Record Open Error

…WARNING (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::3840) [ERROR]<ldt_common_2014_12_20.A:openSubRec()> SubRec Open Failure: Digest(…) Parent Digest(NULL)

…WARNING (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::2567) [ERROR]<ldt_common_2014_12_20.A:openSubRec()> SubRec Open Failure: Digest(…) Parent Digest(…)

…WARNING (udf): (/opt/aerospike/sys/udf/lua/ldt/lib_llist.lua::1906) [ERROR]<ldt_common_2014_12_20.A:openSubRec()> SubRec Open Failure: Digest(…) Parent Digest(NULL)

Any updates on this?


#19

@caz_nagai, @dougluce, @poznia, @Anudeep_Sharma and @mrbar42:

Thank you for posting about LDTs in our forum. Please see the LDT Feature Guide for current LDT recommendations and best practices.


#20

@caz_nagai, @dougluce, @poznia, @Anudeep_Sharma and @mrbar42:

Effective immediately, we will no longer actively support the LDT feature and will eventually remove the API. The exact deprecation and removal timeline will depend on customer and community requirements. Instead of LDTs, we advise that you use our newer List and SortedMap APIs, which are now available in all Aerospike-supported clients at the General Availability level. Read our blog post for details.