[Bug] Actual limits of an LLIST (AER-4300)


#1

Problem Summary

Hi!

One of our LLISTs became too large to scan it so we decided to switch to using find_from instead. This is where I’ve noticed a weird behavior of find_first and find_from functions.

We store numbers there. find_first returned bigger numbers than I could get using find_from calls. Meanwhile, find_from failed to fetch some items inside the range of first and last item.

How to Reproduce

I decided to reproduce the issue locally using Aerospike Community Edition build 3.5.15 inside a Vagrant box.

Storing 1k items in a LLIST (compact list) yielded expected results. 10k records too.

Then I created two more testing subjects:

  1. A list of map[string]int{“key”: -i, “value”: i} with 44,904,583 records inside
  2. A list of int(-i) with 52,154,633 records inside

Both lists were filled with items in parallel (by two processes) for 20 hours. Items to each list were added sequentially in a single thread. Starting value for each list was 100,000,000.

List #1

Below are results from list #1:

find_first(1): -100,107,363 (expected: -144,904,583)
find_last(1): -100,000,000 (correct)
find_from(-144,000,000, 1): UDF Execution Timeout
find_from(-101,000,000, 1): UDF Execution Timeout
find_from(-100,107,363, 1): UDF Execution Timeout
find_from(-100,107,300, 1): UDF Execution Timeout
find_from(-100,100,000, 1): UDF Execution Timeout
find_from(-100,050,000, 1): UDF Execution Timeout
find_from(-100,010,000, 1): -100,010,000 (correct)

This is a part of what llist.config returned to me for list #1:

{
    “SUMMARY”: “LList Summary”,
    “PropRecType”: 1,
    “PropMagic”: “MAGIC”,
    “NodeCount”: 0,
    “StoreState”: “R”,
    “PropParentDigest”: null,
    “PropLdtType”: “LLIST”,
    “PropVersion”: 2,
    “PropItemCount”: 44904583,
    “RootKeyList”: [-128426095, -121360047, -114293999, -107227951, -107200681, -103587361],
    “PropSubRecCount”: 149883,
    “PropSelfDigest”: null,
    “TreeLevel”: 4,
    “LeafCount”: 148757,
    “PageSize”: null
}

List #2

Below are results from list #2:

find_first(1): -100,322,444 (expected: -152,154,633)
find_last(1): -100,000,000 (correct)
find_from(-152,100,000, 1): UDF Execution Timeout
find_from(-101,100,000, 1): UDF Execution Timeout
find_from(-100,322,444, 1): UDF Execution Timeout
find_from(-100,322,400, 1): UDF Execution Timeout
find_from(-100,300,000, 1): UDF Execution Timeout
find_from(-100,200,000, 1): UDF Execution Timeout
find_from(-100,100,000, 1): UDF Execution Timeout
find_from(-100,050,000, 1): -100,050,000 (correct)

This is a part of what llist.config returned to me for list #2:

{
    “SUMMARY”: “LList Summary”,
    “PropRecType”: 1,
    “PropMagic”: “MAGIC”,
    “NodeCount”: 0,
    “StoreState”: “R”,
    “PropParentDigest”: null,
    “PropLdtType”: “LLIST”,
    “PropVersion”: 2,
    “PropItemCount”: 52154633,
    “RootKeyList”: [-121625632, -110773837],
    “PropSubRecCount”: 57965,
    “PropSelfDigest”: null,
    “TreeLevel”: 4,
    “LeafCount”: 57529,
    “PageSize”: null
}

Conclusion

We are trying to store really long lists (tens of millions of items) in LLISTs but when a list gets somewhat large it fails to work as expected. Currently our system is partially broken because of this issue and I see no good way to fix it.

Please help me shed some light on the issue and please tell me if I can provide any more details.

BR, Gregory


Actual limits of a SortedMap CDT
#2

localhots,

This seems to be something to do with negative values. Looking into it. Is it possible to try with non-negative key values

– R


#3

I tried filling a list with 3M positive numbers and seems like you’re right, all functions behave as expected. A list of 500k negative numbers should be enough to reproduce the problem.


[Resolved] Cause of AerospikeError: -2 & message "invalid size for buffer:"? [Bug]
#4

Thanks for reporting this, will fix this.

FYI, this is being tracked with JIRA AER-4300.

– R


#5

How were you running you llist operations

  1. UDF
  2. Client (Which client)
  3. AQL

– R


#6

Using Go client http://godoc.org/github.com/aerospike/aerospike-client-go


#7

Hi!

Any updates on the issue? I see it was not fixed in two past releases. Is it scheduled to be fixed in one of upcoming releases?

Gregory


#8

We unfortunately won’t have much chance to work on this in the immediate future. The LLIST code is available. If anyone can spot an easy fix, that would be great!


#9

@localhots,

Thank you for posting about LDTs in our forum. Please see the LDT Feature Guide for current LDT recommendations and best practices.


#10

@localhots,

Effective immediately, we will no longer actively support the LDT feature and will eventually remove the API. The exact deprecation and removal timeline will depend on customer and community requirements. Instead of LDTs, we advise that you use our newer List and SortedMap APIs, which are now available in all Aerospike-supported clients at the General Availability level. Read our blog post for details.