How to implement Time To Live (TTL) on LDT elements: example (UDF for removing expired items from an LLIST)


#1

Hi

I have been asked many times how to implement a TTL on elements in a Large List (LDT). I finally got around to writing an example.

You will find it at https://github.com/aerospike/LDT-TTL-example

I hope this helps.

Peter


#2

Thanks for sharing your implementation, @helipilot50.

If I understand correctly, llist.scan()–which your solution uses–copies the entire list into lua. So this method will either be not very resource-efficient (entire llist loaded at once) or may not work if the llist is too large.

I think a pagination-based solution will work well, using functions like find_first, find_last, find_from. If I come up with working code I’ll share it.


#3

Following up on the previous post, I wrote a pagination-based UDF for removing expired items from a LLIST. It works, but is not perfect.

Explanation:
Assumes LLIST is a large list of maps, such as:

[
    {"key":100, "time":1444546300000, "value":"value1"}, 
    {"key":101, "time":1444547000000, "value":"value2"}, 
    ...
]

Function:

llist_expire_by_timestamp(rec, bin, time_key, min_time, step_size)

time_key = The map key whose value corresponds to the item’s timestamp
min_time = Minimum timestamp. Entries with timestamps smaller than this value will be expired (deleted)
step_size = Llist page size. Tune this to balance memory usage vs. iterations required

Example usage:

llist_expire_by_timestamp(rec, bin, "time", 1444546305616, 10000)

Code:

function llist_expire_by_timestamp(rec, bin, time_key, min_time, step_size)  
    -- Input check
    if not type(time_key) == 'string' then
        return false
    elseif not type(min_time) == 'number' then
        return false
    elseif not type(step_size) == 'number' then
        return false
    end

    -- Record check
    if not aerospike:exists(rec) then
        return true
    end

    local llist_size = llist.size(rec, bin)
    if llist_size == nil then
        return true
    elseif llist_size == 0 then
        return true
    end
    
    -- Setup for paging through llist
    local key_cursor = llist.find_first(rec, bin, 1)[1]['key']
    local end_reached = false
    local allDeletesOk = true

    -- Page through llist. Delete expired items
    while not end_reached do
        local deleteList = list.create(step_size)
        local result = llist.range(rec, bin, key_cursor, nil, step_size) 
        
        -- Process entire page EXCEPT the final item 
        local scan_range = list.size(result) - 1
        for i=1, scan_range, 1 do
            if result[i][time_key] < min_time then
                list.append(deleteList, result[i]['key'])
                info("will delete " .. tostring(result[i]))
            end
        end

        if list.size(result) == 1 then
            -- Last iteration reached. Process page's final item
            end_reached = true
            if result[scan_range+1][time_key] < min_time then
                info("will delete* " .. tostring(result[scan_range+1]['key']))
                list.append(deleteList, result[scan_range+1]['key'])
            end
        else 
            key_cursor = result[list.size(result)]['key']
        end

        if list.size(deleteList) > 0 then
            -- Delete expired items from current iteration
            info('preparing to delete')
            local deleteResult = llist.remove_all(rec, bin, deleteList)
            if not deleteResult == 1 then
                allDeletesOk = false
            end
        end
    end
    return allDeletesOk
end

#4

Hi

This is quite a nice approach. Its advantages outweigh its limitations

Good job

Peter


#5

@ronny1204,

Thank you for posting about LDTs in our forum. Please see the LDT Feature Guide for current LDT recommendations and best practices.


#6

@ronny1204,

Effective immediately, we will no longer actively support the LDT feature and will eventually remove the API. The exact deprecation and removal timeline will depend on customer and community requirements. Instead of LDTs, we advise that you use our newer List and SortedMap APIs, which are now available in all Aerospike-supported clients at the General Availability level. Read our blog post for details.