Setting time-to-live(ttl)

Knowledge Base General
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Setting time-to-live(ttl)

Outlined in this article are a few methods to set a record’s expiration, also known as time-to-live(ttl).

Refer to the FAQ on TTL article for details regarding TTL usage in Aerospike.

Records that do not have an expiration set (ttl = 0) will not be eligible for evictions when the capacity of a node increases and breaches the disk (high-water-disk-pct) or memory (high-water-memory-pct) high water mark. Based on the use case, it may be useful to selectively set the ttl of records in order to enable them to participate in potential evictions and avoid running out of memory or disk space.

Getting and Setting Record time-to-live(ttl) values

Following are general examples that may not suit specific use cases. It is highly recommended to thoroughly test the provided input in Dev/QA environments prior to enacting this information against a production environment.

UDF

A UDF (User Defined Function) can be used to accomplish the task of setting ttl.

Refer to: http://www.aerospike.com/docs/udf/api/record.html

record.ttl()

  • Get the time-to-live (ttl) of a record.

record.set_ttl()

  • Set the time-to-live (ttl) of a record. When the record is updated this ttl will take effect.
  • KNOWN ISSUE: Reading the ttl within the same UDF transaction context, does not return the updated ttl, even though set_ttl() and update() are called. A subsequent transaction call will correctly return the ttl.

UDF example code

This udf will cap the records that have a ttl greater than 10 years:

aql> execute ttl.change_ttl() on demo.test

function change_ttl(rec)
  local rec_ttl = record.ttl(rec)
  if rec_ttl > 315360000 then
    record.set_ttl(rec, 315360000)
    aerospike:update(rec)
  end
end

Refer to the Modify TTL using UDF article for more details on updating the ttl of a record using a UDF.

Python script example

Here’s a python script (you can grab the aerospike client with pip install aerospike) that will fill two kinds of records into a test set:

  • Half with a -1 ttl or also known as a ‘never expire’ ttl.
  • Half with a 1 hour(3600 seconds) ttl.
# coding: utf-8
from __future__ import print_function
import aerospike

config = { 'hosts': [('127.0.0.1', 3000)] }
client = aerospike.client(config).connect()

i = 1
while i < 100:
    client.put(('test','foo',i), {'i':i}, meta={'ttl':-1}, policy={'exists': aerospike.POLICY_EXISTS_CREATE_OR_REPLACE})
    i = i + 1

i = 101
while i < 200:
    client.put(('test','foo',i), {'i':i, 'x':i}, meta={'ttl':3600})
    i = i + 1

Run the python script.

Check that the python script generated the expected records.

For example in AQL do the following:

aql> select * from test.foo
+——---+---—-+
| i   | x   |
+——---+---—-+
| 44  |     |
| 183 | 183 |
| 134 | 134 |
| ... | ... |
| 139 | 139 |
| 3   |     |
| 128 | 128 |
| 20  |     |
+——---+---—-+
198 rows in set (0.038 secs)

For the purpose of this example, the following Lua module should be saved as ttlpurge.lua.

function purge_ttl_zero(rec)
if record.ttl(rec) ~= 0 then
aerospike:remove(rec)
end
end

Now register the Lua module in AQL (or any other way you prefer).

$ aql
aql> register module './ttlpurge.lua'
OK, 1 module added.

Verify the lua module was registered.

aql> show modules
+--------------------------------------------+----------------+-------+
| hash                                       | module         | type  |
+--------------------------------------------+----------------+-------+
| "67c078987314163f24b9bb9627c6879679f5fcbf" | "ttlpurge.lua" | "lua" |
+--------------------------------------------+----------------+-------+
1 row in set (0.003 secs)

Execute the record UDF against the test.foo set

aql> execute ttlpurge.purge_ttl_zero() on test.foo
OK, scan job (1819164896042454307) created.

Verify the change:

aql> select * from test.foo
+-----+
| i   |
+-----+
| 44  |
| ... |
| 3   |
| 20  |
+-----+
99 rows in set (0.037 secs)

This is as expected.

At this point you can modify and run this script against your namespace or set of choice.

AQL example

AQL always shows the TTL of a record rather than its void time (which is the absolute expiration time stored on the server side). Let’s walk through a couple of example and start by assuming the following server side configuration:

Namespace config

namespace test {
    replication-factor 2
    memory-size 4G
    default-ttl 0  # use 0 to never expire/evict.

    storage-engine device {
    file /opt/aerospike/test.dat
    }

At this point, let’s check what AQL set the TTL to for a potential transaction on a record:

aql> get record_ttl
RECORD_TTL = 0

This means the AQL client is not setting a specific TTL on its side and is instead using the default-ttl specified in the namespace configuration file. In this example it would be 0 meaning the record will never expire.

aql> insert into test.demo (PK, bin1) values (1, 10)
OK, 1 record affected.
aql> select * from test.demo where PK=1

[
    [
        {
          "edigest": "N/A",
          "ttl": -1,
          "gen": 8,
          "bins": {
            "bin1": 10,
            "0": 465128201
          }
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

When the newly inserted record is fetched, the TTL is calculated and printed as -1 on AQL (similar to any other client actually). This is expected and by design. -1 on the client side means the record’s TTL is infinite or that it will never expire (and will also not be eligible for evictions).

Let’s now update this record while specifying a TTL, in this example, 10000.

aql>  set record_ttl 10000
RECORD_TTL = 10000
aql>
aql> get record_ttl
RECORD_TTL = 10000
aql> insert into test.demo (PK, bin1) values (1, 10)

[
    [
        {
          "Status": 0,
          "Message": "1 record affected."
        }
    ]
]
aql> select * from test.demo where pk=1

[
    [
        {
          "edigest": "N/A",
          "ttl": 9991,
          "gen": 10,
          "bins": {
            "bin1": 10,
            "0": 465128201
          }
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]
aql> select * from test.demo where pk=1

[
    [
        {
          "edigest": "N/A",
          "ttl": 9983,
          "gen": 10,
          "bins": {
            "bin1": 10,
            "0": 465128201
          }
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

When this record is read through AQL, the TTL now shows 9983. This happened 17 seconds after writing the record. Again AQL here interprets and translate the void time set on the server to display the expiration as a TTL.

Notes

The following document breaks down the ttl settings behavior for record removal:

The following FAQ describes the TTL behavior in more details:

Latency Warning

When setting or updating the ttl value using a script, many deletes could be generated. This could impact the performance of the cluster as delete transactions would be competing with client application transactions over fabric (inter node communication layer). Large number of deletes could also trigger increase defragmentation activity which could also impact storage i/o performance.

Keywords

ttl update lua python udf aql record set

Timestamp

02/23/2019

1 Like