Changes to Default TTL Not Effecting Existing Data

Operations Tuning
1

Hi,

I changed the default ttl on a 5-node cluster from 1095D to 15D. When I look at the ttl histogram though it looks like the bucket width is still set to 1095 days, meaning I have data that is still not set to expire for a very long time. I expected that changing the ttl would mean all data that was older than 15 days (and had not been touched, i.e. the generation had not been updated) would expire and I could reclaim a bunch of disk space. That has not happened.

My configs have this entry:

namespace prod { replication-factor 1 memory-size 30G default-ttl 15d # 30 days, use 0 to never expire/evict.

But the histogram shows this:

ip-[redacted] returned: prod:ttl=100,945327,34,232,50805372,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4826,0,0,1105,83,681,1323,1614,2464,1983,1017,2648,1541,2932,2213,2770,2073,2303,2258,2205,2269,2993,148475;

ip-[redacted] returned: prod:ttl=100,945327,43,238,50299963,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4785,0,0,1127,86,681,1305,1626,2465,1908,1002,2596,1574,3084,2185,2620,2104,2319,2193,2142,2184,2919,147407;

ip-[redacted] returned: prod:ttl=100,945327,43,194,49324719,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4659,0,0,1019,75,653,1302,1596,2515,1789,970,2574,1511,2984,2098,2805,2142,2183,2090,2243,2145,2871,144381;

ip-[redacted] returned: prod:ttl=100,945327,23,217,51636245,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4743,0,0,1141,76,654,1391,1600,2451,1918,1041,2591,1586,3045,2226,2746,2131,2335,2356,2275,2299,2935,149478;

ip-[redacted] returned: prod:ttl=100,945327,0,0,51064579,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5209,0,0,1138,93,764,1452,1737,2700,1958,1058,2726,1619,3217,2277,2993,2308,2489,2383,2287,2323,3092,158776;

Is there a way to force Aerospike to expire this data?

Thank you.

2

Is there any documentation that led you to this expectations for default-ttl?

You would need to run a scan looking for records with TTLs above your threshold and delete them. Since you aren’t interested in the records data (just meta-data) you should configure your scan to only return the metadata.

3

Nope, I totally made it up. I found a really good, detailed explanation of ttls and evictions here https://discuss.aerospike.com/t/records-ttl-and-evictions/737, but couldn’t locate something similar related to expirations. It seemed like a logical expectation, but now that I think about your question I’m putting together that the TTL is part of the metadata for a record so unless Aerospike would have to go through and change the generation on every pre-existing record, which is probably not very efficient, so I guess it makes sense that they would not have a feature to do this—automatically, at least.

Thank you.

4

The configuration parameter default-ttl only impacts new writes, and that is limited to new records not being written with explicit ttl values. Existing records will not be impacted through modification of this configuration parameter, thus your existing records would still be expiring at 1095D and not expiring as hoped at 15D.

The solution is to effect a Scan touch operation and update the TTL This can be accomplished using a User-Defined Function (UDF) with the Lua Programming Language or programmatically(using our supported Client Libraries).

To accomplish this using the Lua Programming Language see: http://www.aerospike.com/docs/udf/developing_lua_modules.html

The following article contains a verify basic example of how to update the ttl see: How to 'touch' record on UDF

NOTE: Make sure you do not enable xdr-ship-bins (Configuration Reference | Aerospike Documentation) as this would then only ship the changed bins.

This document outlines the process to execute the UDF: http://www.aerospike.com/docs/tools/aql/udf_management.html

In the case of needing to kill_scan more details can be found in the AQL docs under Query and Scan Management see: http://www.aerospike.com/docs/tools/aql/query_scan_management.html

To deepen your understanding on UDF see: User-Defined Functions (UDF) Development Guide | Developer

Scan touch can also be performed programmatically, using our supported Client Libraries

For Example, the Java Client API allows you to perform operations on records. http://www.aerospike.com/docs/client/java/usage/kvs/multiops.html Here is an example of Scan Records code http://www.aerospike.com/docs/client/java/usage/scan/scan.html Here is the API Reference including information on the touch method. aerospike-client 7.2.0 javadoc (com.aerospike)

Here is another example using C Client API on Touching a Record http://www.aerospike.com/docs/client/c/usage/kvs/multiops.html#touching-a-record

5

Additionally, just changing the config file in an already up and running cluster will not change the default ttl setting for future records. It is a dynamic parameter, ie it can be changed on the fly but you must use asinfo -v via asadm to effect it on all the active nodes. If you change in the config file, it will only be read and implemented when you restart the cluster.

asadm [ENTER]
Admin>asinfo -v 'set-config:context=namespace;id=test (or whatever your namespace is);default-ttl=15d'
6

@Avi - exists is a boolean return value operation that examines only the primary index in RAM and returns. Does not affect the record in any way.