Setting time-to-live(ttl)


#1

Setting time-to-live(ttl)

Outlined in this article are a few methods to set a record’s expiration, also known as time-to-live(ttl).

Records that do not have an expiration set (ttl = 0) will not be eligible for evictions when the capacity of a node increases and breaches the disk (high-water-disk-pct) or memory (high-water-memory-pct) high water mark. Based on the use case, it may be useful to selectively set the ttl of records in order to enable them to participate in potential evictions and avoid running out of memory or disk space.

Getting and Setting Record time-to-live(ttl) values

Following are general examples that may not suit specific use cases. It is highly recommended to thoroughly test the provided input in Dev/QA environments prior to enacting this information against a production environment.

Finding records time-to-live(ttl)

The following article explains how to find records with “0 ttl” using the asadm tool or all values, including “-1 ttl”, using asbackup or UDF.

See: Find records with TTL 0

JAVA

This following code can be modified to set record ttl values using Java:

Refer to: https://github.com/aerospike/delete-set Link to Java API reference: https://www.aerospike.com/apidocs/java/

UDF

A UDF (User Defined Function) can be used to accomplish the task of setting ttl.

Refer to: http://www.aerospike.com/docs/udf/api/record.html

record.ttl()

  • Get the time-to-live (ttl) of a record.

record.set_ttl()

  • Set the time-to-live (ttl) of a record. When the record is updated this ttl will take effect.
  • KNOWN ISSUE: Reading the ttl within the same UDF transaction context, does not return the updated ttl, even though set_ttl() and update() are called. A subsequent transaction call will correctly return the ttl.

UDF example code

aql> execute ttl.change_ttl() on demo.test
function change_ttl(rec)
  local rec_ttl = record.ttl(rec)
  if rec_ttl > 315360000 then
    record.set_ttl(rec, 315360000)
    -- for the TTL update to happen a bin must be 'dirty'
    -- set the first bin back with the value of the first bin
    -- no changes will occur other than the TTL change, but the
    -- aerospike:update(rec) will actually execute
    local bin_names = record.bin_names(rec)
    local do_nothing = rec[bin_names[1]]
    rec[bin_names[1]] = do_nothing
    aerospike:update(rec)
  end
end

Link to using UDF with AQL: http://www.aerospike.com/docs/tools/aql/udf_management.html

Python script example

Here’s a python script (you can grab the aerospike client with pip install aerospike) that will fill two kinds of records into a test set:

  • Half with a -1 ttl or also known as a ‘never expire’ ttl.
  • Half with a 1 hour(3600 seconds) ttl.
# coding: utf-8
from __future__ import print_function
import aerospike

config = { 'hosts': [('127.0.0.1', 3000)] }
client = aerospike.client(config).connect()

i = 1
while i < 100:
    client.put(('test','foo',i), {'i':i}, meta={'ttl':-1}, policy={'exists': aerospike.POLICY_EXISTS_CREATE_OR_REPLACE})
    i = i + 1

i = 101
while i < 200:
    client.put(('test','foo',i), {'i':i, 'x':i}, meta={'ttl':3600})
    i = i + 1

Run the python script.

Check that the python script generated the expected records.

For example in AQL do the following:

aql> select * from test.foo
+——---+---—-+
| i   | x   |
+——---+---—-+
| 44  |     |
| 183 | 183 |
| 134 | 134 |
| ... | ... |
| 139 | 139 |
| 3   |     |
| 128 | 128 |
| 20  |     |
+——---+---—-+
198 rows in set (0.038 secs)

For the purpose of this example, the following Lua module should be saved as ttlpurge.lua.

function purge_ttl_zero(rec)
if record.ttl(rec) ~= 0 then
aerospike:remove(rec)
end
end

Now register the Lua module in AQL (or any other way you prefer).

$ aql
aql> register module './ttlpurge.lua'
OK, 1 module added.

Verify the lua module was registered.

aql> show modules
+--------------------------------------------+----------------+-------+
| hash                                       | module         | type  |
+--------------------------------------------+----------------+-------+
| "67c078987314163f24b9bb9627c6879679f5fcbf" | "ttlpurge.lua" | "lua" |
+--------------------------------------------+----------------+-------+
1 row in set (0.003 secs)

Execute the record UDF against the test.foo set

aql> execute ttlpurge.purge_ttl_zero() on test.foo
OK, scan job (1819164896042454307) created.

Verify the change:

aql> select * from test.foo
+-----+
| i   |
+-----+
| 44  |
| ... |
| 3   |
| 20  |
+-----+
99 rows in set (0.037 secs)

This is as expected.

At this point you can modify and run this script against your namespace or set of choice.

AQL example

AQL always shows the TTL of a record rather than its void time (which is the absolute expiration time stored on the server side). Let’s walk through a couple of example and start by assuming the following server side configuration:

Namespace config

namespace test {
    replication-factor 2
    memory-size 4G
    default-ttl 0  # use 0 to never expire/evict.

    storage-engine device {
    file /opt/aerospike/test.dat
    }

At this point, let’s check what AQL set the TTL to for a potential transaction on a record:

aql> get record_ttl
RECORD_TTL = 0

This means the AQL client is not setting a specific TTL on it’s side and is instead using the default-ttl specified in the namespace configuration file. In this example it would be 0 meaning the record will never expire.

aql> insert into test.demo (PK, bin1) values (1, 10)
OK, 1 record affected.
aql> select * from test.demo where PK=1

[
    [
        {
          "edigest": "N/A",
          "ttl": -1,
          "gen": 8,
          "bins": {
            "bin1": 10,
            "0": 465128201
          }
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

When the newly inserted record is fetched, the TTL is calculated and printed as -1 on AQL (similar to any other client actually). This is expected and by design. -1 on the client side means the record’s TTL is infinite or that it will never expire (and will also not be eligible for evictions).

Let’s now update this record while specifying a TTL, in this example, 10000.

aql>  set record_ttl 10000
RECORD_TTL = 10000
aql>
aql> get record_ttl
RECORD_TTL = 10000
aql> insert into test.demo (PK, bin1) values (1, 10)

[
    [
        {
          "Status": 0,
          "Message": "1 record affected."
        }
    ]
]
aql> select * from test.demo where pk=1

[
    [
        {
          "edigest": "N/A",
          "ttl": 9991,
          "gen": 10,
          "bins": {
            "bin1": 10,
            "0": 465128201
          }
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]
aql> select * from test.demo where pk=1

[
    [
        {
          "edigest": "N/A",
          "ttl": 9983,
          "gen": 10,
          "bins": {
            "bin1": 10,
            "0": 465128201
          }
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

When this record is read through AQL, the TTL now shows 9983. This happened 17 seconds after writing the record. Again AQL here interprets and translate the void time set on the server to display the expiration as a TTL.

Notes

The following document breaks down the ttl settings behavior for record removal:

http://www.aerospike.com/docs/operations/manage/storage#setting-time-to-live-ttl-for-data-expiration

Latency Warning

When setting or updating the ttl value using a script, many deletes could be generated. This could impact the performance of the cluster as delete transactions would be competing with client application transactions over fabric (inter node communication layer). Large number of deletes could also trigger increase defragmentation activity which could also impact storage i/o performance.

Keywords

ttl update lua java python udf aql record set

Timestamp

09/19/2017