Allow duplicate keys in Large List in aerospike server 3.5.12


#1

Prior to aerospike server version 3.5.4 the duplicate keys can be allowed in Large List by overriding the llist settings in udf and passing that module while creating the large list as suggested in How to avoid LDT-Unique Key or Value Violation on Ordered List.

In future releases, the distribution does not comes with /opt/aerospike/sys/udf/lua/ldt/settings_llist.lua. Also the java client library have deprecated the API which takes userModule as argument and simply ignores the module while creating the large list.

Is it still possible to enable duplicate keys in the Large List in latest version and if so what are the steps to configure Large List to allow duplicate keys.


#2

Also while using aerospike version 3.5.4 and enabling non-unique keys in the Large List we are facing the issue as mentioned in Error LLIST with non unique keys.

After inserting 6 records with same key, we start receiving the following errors com.aerospike.client.AerospikeException: Error Code 1400: /opt/aerospike/sys/udf/lua/ldt/lib_llist.lua:4380 LDT-Internal Error

The issue is always reproducible.

Another update on the issue:

The same error is coming even after inserting some records with different(unique) keys in the large list ldt if the duplicate keys were enabled for the large list. com.aerospike.client.AerospikeException: Error Code 1400: /opt/aerospike/sys/udf/lua/ldt/lib_llist.lua:4380 LDT-Internal Error


#3

Shalabh,

Duplicate keys in NOT yet supported in the LLIST. Behavior is not defined if you enable it …

– R


#5

This feature would be really cool, if production-proof. Our current work around is to use a 64-bit timestamp with microseconds resolution which should work fine even a trillion times, however it feels bad to use something that is not guaranteed to work from a logical standpoint - and it’s only possible due to the event-nature of our use case (we initially wanted to use a 32-bit unix timestamp but saw that Large Ordered Lists will use 64-bit anyways).

In our data scheme we also use lists that are updated only by appending (think of it like events appended to a log) quite often, where the ordering logic is not really needed (it should not hurt performance, does it?). Using a timestamp there is kinda handy in terms of querying ranges later on, but like I said it feels kinda bad.

But maybe there is another option to achieve uniqueness in your use case (e.g. use 32 bits for the key and the other 32-bits get set randomly / auto incremented) and then query with a range from xxxxx_0b00000…000 till xxx_0b111111.111. However, integrated support for non-unique keys would make things a lot easier for developers.

Cheers, Manuel


#6

Manuel,

Thanks for the inputs… I hear you. Will check and see how we can prioritize this.

– R


#7

@shalabh, @ManuelSchmidt:

Thank you for posting about LDTs in our forum. Please see the LDT Feature Guide for current LDT recommendations and best practices.


#8

@shalabh, @ManuelSchmidt:

Effective immediately, we will no longer actively support the LDT feature and will eventually remove the API. The exact deprecation and removal timeline will depend on customer and community requirements. Instead of LDTs, we advise that you use our newer List and SortedMap APIs, which are now available in all Aerospike-supported clients at the General Availability level. Read our blog post for details.


#9

Thanks for the update.