Disk persistance and memory caching?

1

Hi, I am new to aerospike and I am building a server with aerospike.

I have a question about disk storage and memory. I want to know if below scenario is possible with aerospike.

  1. Store data in disk not memory
  2. But, once a data is accessed(write or read) by client, the data leaves on aerospike memory within the ttl.
  3. So, within the ttl, if client access(write or read) the data, aerospike would response from aerospike memory.
  4. if the data in memory accessed within the ttl, the ttl of the data will be extended.

thanks

2

You can easily do this by implementing the logic in the client - nothing native in Aerospike that you can just turn on this behavior.

Option 1 - You will create two namespaces, one on disk with the desired longer TTL, other in RAM with the desired shorter TTL. You will first read in RAM, if record not found, read from disk, and then also write it back to RAM. (Here, you will have to also consider if there can possibly be any consistency issues because each read or write in Aerospike is atomic. Multi-client situation, data updated on disk by another client after one client read it.) If you are able to read successfully from RAM, touch the record to update its TTL. Read and touch operation can be combined in the same transaction. If the data is single bin integer or float, further optimization is possible.

Option 2 - It is also possible to store the same record data both in disk and RAM simultaneously - on disk for persistence, in RAM for faster performance. Con - your entire data set now resides both in disk and RAM. But you can achieve this in a single namespace. The TTL will be same for both stores then. To update TTL on read, you have to essentially do a write (touch), but both can be done in the same transaction.

3

Hi pgupta. Thanks for your detail explaination.

I knew both solutions already, but solution 1 makes the client code very complex, solution 2 could cause out of memory.

I want to access arrospike just one time and aerospike controll the memory as cache and disk as storage.

Please give me idea.

4

You can submit a feature request on this forum - look for the “Feature Request” section. This is not a current feature in Aerospike. Aerospike does have a “data-in-memory” option on a device namespace - what you want is “data-in-memory-on-read” option, and with its own TTL in RAM. Interesting feature, but I think it will be very time consuming to implement since the TTL is at the record level, stored in the Primary Index and evictions and expirations run off it. So associating two TTLs with a record, one for device storage and another for RAM storage - and then expiration and eviction thread treating the two stores differently - this is a significant change. Can it be done? Probably - will have to store the TTL for the RAM copy along with the data and then run a separate expiration process on the RAM data only. This is a major change from my understanding of the architecture - but I am not in Aerospike Engineering! :slight_smile:

5

Thanks again. I will raise “Feature Request” and hope this be possible.

If this is possible on aerospike, aerospike will have killer feature.