LSTACK Storage error


#1

I want to add a jsondata which eventualy grows in size,Which LDT is appropriate for it? I am using LTSTACK where in i am dividing data into chunks and then pushing it on to the stack.? What can i do to put such large data into ldt without losing much speed? And top record update and top record create occurs when byte array chunks are pushed on to the stack where as when i push chunks of string no error occurs.


#2

Hi. Large Stack (LSTACK) is the fastest of the LDTs for two reasons. First, it does not need to do any value comparisons, it just pushes the value onto the stack. Second, most push() calls affect only the Top Record, and not any sub-records, as most push calls affect only the “Hot List”, which resides in the record itself. Only when the Hot List overflows, and a portion of the Hot List is moved off-record to the Warm List, are there any additional I/O operations (and even then, only ONE additional I/O).

All of the LDT inserts will implicitly create the Aerospike Top Record if it does not yet exist.

Currently, the default Hot List size is 100 elements, but if you are going to insert items larger than, say, 1kb, then you should configure the LSTACK for larger size elements (basically, lower the Hot List Size) so that you don’t overflow the Top record storage capacity.

If we notice that most customers tend to store larger objects (over 1kb in size), then it is likely that we’ll change the default value of the Hot List size to be smaller, to accomodate more default usage.

Toby


#3

This is my lstack standard settings

– General LSTACK Parms: ldtMap[T.M_StoreMode] = SM_LIST; ldtMap[T.M_Transform] = nil; ldtMap[T.M_UnTransform] = nil; – LSTACK Data Record (LDR) Chunk Settings: Passed into "Chunk Create" ldtMap[T.M_LdrEntryCountMax] =1000;-- Max # of items in an LDR (List Mode) ldtMap[T.M_LdrByteEntrySize] =81920; – Byte size of a fixed size ByteEntry ldtMap[T.M_LdrByteCountMax] =5242480; – Max # of BYTES in an LDR (binary mode) – Hot Entry List Settings: List of User Entries ldtMap[T.M_HotListMax] = 600; – Max # for the List, when we transfer ldtMap[T.M_HotListTransfer] = 300; – How much to Transfer at a time – Warm Digest List Settings: List of Digests of LSTACK Data Records ldtMap[T.M_WarmListMax] = 1000; – # of Warm Data Record Chunks ldtMap[T.M_WarmListTransfer] = 5000; – # of Warm Data Record Chunks – Cold Directory List Settings: List of Directory Pages ldtMap[T.M_ColdListMax] = 10000; – # of list entries in a Cold dir node ldtMap[T.M_ColdDirRecMax] = 100000; – Max# of Cold DIRECTORY Records end – package.StandardList()

My requirement is,i want to put byte array on to the stack,where each element is of size 3000,(byte[3000]) the range of chunks(byte[3000]) can range max from 1-200max…So for every key record i am having LDT LTSTACK bin. Its still showing top record update error


#4

Hi,

So, I’m guessing that you’ve gone and updated the Lua code yourself. That’s a fairly dangerous thing to do, as the values you’ve chosen probably do not do what you’re thinking they do. The values you’re showing here don’t look right to me.

With these values you will blow out the top record or the sub-record and get record or sub-record write errors.

If you describe exactly what it is you want to achieve, I can give you the correct values for the settings.

Better yet, I’m just about ready to release a new mechanism that takes in user values (object size, object count) and does the appropriate computation for all of the configuration values.


#5

function package.StandardList( ldtMap ) – General LSTACK Parms: ldtMap[T.M_StoreMode] = SM_LIST; ldtMap[T.M_Transform] = nil; ldtMap[T.M_UnTransform] = nil; – LSTACK Data Record (LDR) Chunk Settings: Passed into "Chunk Create" ldtMap[T.M_LdrEntryCountMax] =1000;-- Max # of items in an LDR (List Mode) ldtMap[T.M_LdrByteEntrySize] =3000; – Byte size of a fixed size ByteEntry ldtMap[T.M_LdrByteCountMax] =5242480; – Max # of BYTES in an LDR (binary mode) – Hot Entry List Settings: List of User Entries ldtMap[T.M_HotListMax] = 1000; – Max # for the List, when we transfer ldtMap[T.M_HotListTransfer] = 600; – How much to Transfer at a time – Warm Digest List Settings: List of Digests of LSTACK Data Records ldtMap[T.M_WarmListMax] = 1000; – # of Warm Data Record Chunks ldtMap[T.M_WarmListTransfer] = 5000; – # of Warm Data Record Chunks – Cold Directory List Settings: List of Directory Pages ldtMap[T.M_ColdListMax] = 10000; – # of list entries in a Cold dir node ldtMap[T.M_ColdDirRecMax] = 100000; – Max# of Cold DIRECTORY Records end – package.StandardList()

And i have set writeblock-size to 1048576 I did some changes! However i am not getting exception for writes.WIll see how read operation goes.But in this case write operation is slow


#6

Hi,

Your settings still don’t look appropriate for what (I think) you’re trying to accomplish.

If you look in your settings_lstack.lua file (/opt/aerospike/sys/udf/lua/ldt/), you’ll see settings for various levels of object support. For example, here are settings you would want to use if you’re using 10kb objects:

– ====================================================================== – For Large Objects (around 10kb) we use a small list size – for the Hot List and a small list for the SubRecords (LDR), but not – as small as the list size for JUMBO objects (above). – Package = “ListLargeObject” – ====================================================================== function package.ListLargeObject( ldtMap ) – General LSTACK Parms: ldtMap[T.M_StoreMode] = SM_LIST; ldtMap[T.M_Transform] = nil; ldtMap[T.M_UnTransform] = nil; – LSTACK Data Record (LDR) Chunk Settings: Passed into "Chunk Create" ldtMap[T.M_LdrEntryCountMax] = 20; – Max # of items in an LDR (List Mode) ldtMap[T.M_LdrByteEntrySize] = 0; – Byte size of a fixed size Byte Entry ldtMap[T.M_LdrByteCountMax] = 0; – Max # of BYTES in an LDR (binary mode) – Hot Entry List Settings: List of User Entries ldtMap[T.M_HotListMax] = 20; – Max # for the List, when we transfer ldtMap[T.M_HotListTransfer] = 10; – How much to Transfer at a time – Warm Digest List Settings: List of Digests of LSTACK Data Records ldtMap[T.M_WarmListMax] = 100; – # of Warm Data Record Chunks ldtMap[T.M_WarmListTransfer] = 10; – # of Warm Data Record Chunks – Cold Directory List Settings: List of Directory Pages ldtMap[T.M_ColdListMax] = 100; – # of list entries in a Cold dir node ldtMap[T.M_ColdDirRecMax] = 100; – Max# of Cold DIRECTORY Records end – package.ListLargeObject()

Picking your own values for these settings will likely result in unintended behavior, unless you know exactly what the values mean. Our next release of the configuration documentation will do a better job of explaining it. However, in the meantime, I recommend using the pre-packaged settings that most closely match your application object sizes.


#7

My object size is ranging from 120kb to 4.4mb,for a record for a particular key. I am using only one LDT BIn for a single key My key is any identifier and i am storing transaction history in json form into single bin.What is the appropriate way to accomodate the large json into single bin. so a single record can weigh from 100kb to 4-5MBs So right now i am dividing d record into chunks,but indviduals chunks while retrieval doesnt make any sense for us,our concern is with retrieval of whole chunks of json from LDT bin.


#8

Hi,

You bring up a topic that is something we’ve been discussing internally.

We know that one day we will need a true “Large Bytes” type (basically a file object) that will let people store a large unstructured object. But, until that day, there is a way to simulate that capability with a UDF and LDT.

This is something that anyone can build with their own UDF, although we will probably build it as an example soon.

In a UDF Module, you would implement two functions: get() and put(). To put(), you input the entire object from the client as a byte array. You then cut it into fixed size pieces (say, 16kb pieces) and write those pieces into either LSTACK or LLIST. If you use LLIST you would probably create a map object for each piece (“key” field is sequence number, and “payload” field is the byte array). To get(), you just scan the LDT and read the pieces back into a result byte array, and then deliver that result byte array back to the client.

From the client, who sees just the put() and get() functions, it appears as a whole large object was stored into a bin, and then was retrieved from the bin. The client would have no knowledge that the object was cut into pieces and then reassembled.


#9

I am already dividing it into chunks of 100kb,encoding json data into byte array and dividing the byte array into chunks and while using get() to retrieving all chunks den decoding entire byte array back to json,i am losing the read speed. 1key :record value 100kb chunks


#10

One thing to be aware of. If you use very large pieces (like 100kb) then you need to make sure that your LDT configuration is not set up to cache any data at all in the Top Record.

Picking a smaller “chunk” size would probably give you better behavior.


#11

Hi Toby,thanks for the reply. What can be the optimum settings for lua where lstack can have 80Kb objects,and the records size can exceed 1mb,with optimum list size,so that record wont over flow and it wont throw top record update /top record create error.Waiting for you reply!


#12

Hi, We have overloaded the LSTACK create() and push() calls to allow the “createSpec” parameter (sometimes called “userModule” in documentation) to specify the LDT configuration settings. The “createSpec” parameter can be either a string (which means it names a UDF that performs the configuration on the first LDT insert), or it is a map (which means it either contains a “package” name of a prepackaged setting, or it contains a “Compute” label that takes in user values, such as max Object Size and target Page Size and computes the best settings).

Although the “Compute” option will not be available for another couple releases, the prepackaged setting is something that you already have – and can use. In your case, you would want the “ListJumboObject” package that is in the settings_lstack.lua file.

Assuming your namespace write block size is at 1mb (and not 128kb), these settings will work for you: – ====================================================================== – For Jumbo Objects (around 100kb) we use a much smaller list size – for the Hot List and a smaller list for the SubRecords (LDR). – Package = “ListJumboObject” – ====================================================================== function package.ListJumboObject( ldtMap ) – General LSTACK Parms: ldtMap[LC.StoreMode] = SM_LIST; – LSTACK Data Record (LDR) Chunk Settings: Passed into "Chunk Create" ldtMap[LS.LdrEntryCountMax] = 8; – Max # of items in an LDR (List Mode) – Hot Entry List Settings: List of User Entries ldtMap[LS.HotListMax] = 8; – Max # for the List, when we transfer ldtMap[LS.HotListTransfer] = 4; – How much to Transfer at a time – Warm Digest List Settings: List of Digests of LSTACK Data Records ldtMap[LS.WarmListMax] = 100; – # of Warm Data Record Chunks ldtMap[LS.WarmListTransfer] = 10; – # of Warm Data Record Chunks – Cold Directory List Settings: List of Directory Pages ldtMap[LS.ColdListMax] = 100; – # of list entries in a Cold dir node ldtMap[LS.ColdDirRecMax] = 100; – Max# of Cold DIRECTORY Records end – package.ListJumboObject()

I’ll post an example shortly that shows exactly how you use it in a program.

Toby


#13

Hi, I was just looking at our “package apply” logic and noticed that I temporarily disabled it recently during some refactoring. So, I’ve just now re-enabled it, but in the meantime, the best way to set LDT configuration values is with the “CreateModule” (the UDF form) createSpec.

So, here are the steps: (1) create the “myCreateModule.lua” file (contents shown below). (2) register the UDF (using either code or a tool, such as AQL or ASCLI) (3) name the CreateModule in the createSpec parameter position of the create() or push() call.

Here’s the contents of the myCreateModule.lua file:

– ====================================================================== – Must define a table that holds the functions that will be exported – and used by the LDT instance. – ====================================================================== local userModule = {};

– ====================================================================== – Lua Imports – ====================================================================== – Import the functions we will be using in Large List to perform the – LSTACK LDT configuration. local lstack_settings = require(‘ldt/settings_lstack’);

– ====================================================================== – adjust_settings() – Set this LSTACK for best performance using – + 80 kilobyte objects – ====================================================================== – This is a specially named function “adjust_settings()” that the LDT – configure code looks for. If an “adjust_settings()” function exists – in this module (it’s a Lua Table entry), it will be called, with the – ldtMap as a parameter, to initialize this LDT instance on either a – create call or on the first insert. – ====================================================================== function userModule.adjust_settings( ldtMap )

– Use a modest amount of space for the In-Record Hot List lstack_settings.set_hotlist_max( ldtMap, 4 ); lstack_settings.set_hotlist_transfer( ldtMap, 2 );

– Let the Sub-Records get fairly large (8 * 80kb) lstack_settings.set_ldr_entry_count_max( ldtMap, 8 );

– Warm List Size depends on usage. Be conservative (small) here. lstack_settings.set_warmlist_max( ldtMap, 20 ); lstack_settings.set_warmlist_transfer( ldtMap, 5 );

– Let the cold list grow fairly large. lstack_settings.set_coldlist_max( ldtMap, 100 ); lstack_settings.set_colddir_rec_max( ldtMap, 100 );

end – adjust_settings()

– ====================================================================== – Return the value of this module’s table so that others importing this – module get the table reference. – ====================================================================== return userModule;

– ======================================================================== – ========================================================================


#14

Thanks for the reply. Hey Toby,I have uploaded a json file of approx 4 mb,send me the lua script and changes to be done in settigns_lstack.lua,assuming i have to divide this file into 80kb of chunks. Here is the link of the file: http://expirebox.com/download/c0433c9b31e8a928e8fd18842e3b3251.html Waiting for your reply.


#15

Hi – look at my previous post again. I think that already contains all the information you need.

And, just FYI … I’m creating a new (short-term) function for LLIST that allows you to write a binary BLOB of any size and get it back. We do the “chunking” on the server side. I’m creating this for those customers that basically want file storage (single large put and get) on the database side. Eventually we’ll have a better, faster mechanism for BLOB storage, but this will work as a temporary bridge.

It will look like this: llist.write_binary(binName, byteArray) llist.read_binary(binName)

Is that something you might use?


#16

Cool man! Sure we can try it.Waiting for it.


#17

function package.StandardList( ldtMap ) – General LSTACK Parms: ldtMap[T.M_StoreMode] = SM_LIST; ldtMap[T.M_Transform] = nil; ldtMap[T.M_UnTransform] = nil; – LSTACK Data Record (LDR) Chunk Settings: Passed into "Chunk Create" ldtMap[T.M_LdrEntryCountMax] =2000;-- Max # of items in an LDR (List Mode) ldtMap[T.M_LdrByteEntrySize] =0; – Byte size of a fixed size ByteEntry ldtMap[T.M_LdrByteCountMax] =0; – Max # of BYTES in an LDR (binary mode) – Hot Entry List Settings: List of User Entries ldtMap[T.M_HotListMax] = 1000; – Max # for the List, when we transfer ldtMap[T.M_HotListTransfer] = 600; – How much to Transfer at a time – Warm Digest List Settings: List of Digests of LSTACK Data Records ldtMap[T.M_WarmListMax] = 1000; – # of Warm Data Record Chunks ldtMap[T.M_WarmListTransfer] = 5000; – # of Warm Data Record Chunks – Cold Directory List Settings: List of Directory Pages ldtMap[T.M_ColdListMax] = 10000; – # of list entries in a Cold dir node ldtMap[T.M_ColdDirRecMax] = 100000; – Max# of Cold DIRECTORY Records end – package.StandardList()

This is my settings for standard list.Device overloaded after inserting 1,60000 records Waiting for flile storage data type settings!


#18

ok – I created the LLIST.read_bytes() and LLIST.write_bytes() functions as a proof of concept, but after further review it became obvious that this is not something that should be offered under the LLIST umbrella. It really needs to be under the umbrella of a new type (i.e. the Large Bytes: LBYTES) type. So, we’re doing that, but, as usual, it has to make its way thru Testing and QA before it can appear on the outside. So, stay tuned. It will be out soon, but just not immediately.

Thanks for your patience.


#19

The numbers you’re using in your LMAP configuration package will not work (they are outside the proper ranges).

Just like the previous LSTACK example – tell me your LMAP configuration parameters: () Your ave object size () Your max object size () Your ave Name (Key) size () Your ave Name (Key) size () Your average (expected) Object count () Your Max (expected) Object count

and I’ll give you the numbers for setting up your LMAP package. And, just a reminder, the automatic configurator will be out soon. It will take these same numbers and automatically configure your LDT appropriately.

Note that you can leave any of those settings as the default, but if your key, object or count sizes are significantly different from the default settings, then you will not get optimal performance or storage efficiency.


#20

I have average record size of 3MB & min is 1MB and Max is 5 MB and can increase,I have divided it into chunk of objects of 80KB size, My average key size here is 100bytes, My object count can range from 1-200max,If object size exceeds 5mb,i will assemble those chunks it into large object size.My concern is to retrieve all objects at once,since my chunking json data,and i want all chunks of data from LSTACK.