In one of our applications we are using a bytes array in LUA to manage an array of key,expire pairs. Pairs can be modified and can expire.
I’m wondering if a developer can give me some insight into which would be the best strategy here.
One thing we do is always set the expire of the whole record to the longest expire value from our key,expire pairs. This way when the oldest one expires we can let aerospike delete the whole record.
But one issue is that when certain pairs are update frequently with new expire values some of the older pairs that have long expired stay in the array.
One way to handle this would be for every modify operation to loop through the whole array and only copy the pairs that haven’t expired yet to a new bytes array. This is the strategy we are using now. This is a simplified version of the code:
function add(r, cid, per, now) local found = false local b = bytes(0) if not aerospike:exists(r) then aerospike:create(r) record.set_ttl(r, per) else local ttl = record.ttl(r) -- If the whole record is expired don't use it at all. -- 2147483648 is because we are using an old version of aerospike which had this bug. if ttl == 0 or ttl > 2147483648 then record.set_ttl(r, per) else local c = r["f"] local csize = bytes.size(c) -- Loop over the old data, copy what hasn't expired yet. for i=1,csize,8 do local expire = bytes.get_int32_le(c, i+4) if expire > now then if bytes.get_int32_le(c, i) == cid then found = true bytes.append_int32_le(b, cid) bytes.append_int32_le(b, now + per) else bytes.append_bytes(b, bytes.get_bytes(c, i, 8), 8) end end end if ttl < per then record.set_ttl(r, per) end end end if found == false then bytes.append_int32_le(b, cid) bytes.append_int32_le(b, now + per) end r["f"] = b aerospike:update(r) return 0 end
My question is; is this the most optimal way to do this in aerospike, or are some optimizations possible?
I don’t know how after the UDF the bytes are actually committed to the table. Would allocating a new bytes array every time cause a lot of fragmentation?
Anyone have any other suggestions on how to improve this?