How to Update record without disturbing the initial set ttl


#1

Hi, I am using Aerospike 3.15.0.1 with Java 4.0.8 client. Below is my usecase.

If Record is getting created: put record with ttl Else: increment and get without disturbing the initially set ttl

To achieve this I am using below code snippet and takes two IO hits if I want to update without changing the ttl.

My query is, how to achieve this in single transaction or single IO hit?

for(int k=this.start; k<=(this.end); k++){
    Key key = new Key("mem", null, k);
    try{
        writePolicy.expiration = 300;
        writePolicy.recordExistsAction = RecordExistsAction.CREATE_ONLY;
        bin1 = new Bin(null, k);
        client.put(writePolicy, key, bin1);
      }catch(AerospikeException e){
          if(e.getResultCode() == ResultCode.KEY_EXISTS_ERROR){
              writePolicy.expiration = -2;
              writePolicy.recordExistsAction = RecordExistsAction.UPDATE_ONLY;  
              bin1 = new Bin(null, 1);
              Record record = client.operate(writePolicy, key, Operation.add(bin1), Operation.get());
          }
    }
}

Appreciate your help. Thanks in Advace.


#2

That should work. Whats the question?


#3

May want to swap the ordering, assume you are updating and if the record doesn’t exist create (assuming more updates than creates).


#4

Have you declared this namespace - mem - to have only single-bin records? I seen bin name as null.


#5

All,

My query is, how to achieve this in single transaction for single IO hit?

At present, it takes 2 IO hits on updates.


#7

Yes. This is a single bin namespace.


#8

Majority of transaction are one time inserts. Hence wrote this way. Anyways thank you for the suggestions.


#9

My query is, how to achieve this in single transaction for single IO hit?

At present, it takes 2 IO hits on updates.


#10

This is the best the current client protocol supports.


#11

You can use a Record UDF and move your logic to server node. In the UDF, use the exists() api to check if the record exists, if it doesn’t, call aerospike:create() else call aerospike:update(). (Pass the k argument to the record udf. ) This will allow you to create/update in one i/o hit from the client to the server.

This topic has some relevant code that can help you get started: Adding Key and TTL when creating a record using Record UDF


#12

@pgupta would this be correct?

function create_or_update_record (rec,bin1,bin2)
  if( not aerospike:exists( rec ) ) then
   rec['bin1']=bin1
   rec['bin2']=bin2
   return aerospike:create(rec);
 else
   rec['bin1']=bin1
   rec['bin2']=bin2
   record.set_ttl(rec, -2)
   return aerospike:update(rec);
 end
end

#13

Yes, that works. Just tested it via aql.

$ aql
aql> register module './c_or_u.lua'
aql> set output json
aql> set record_print_metadata true
aql> EXECUTE c_or_u.create_or_update_record(2,3) ON test.demo where pk='k1'

– Created first version of k1

– Create second version of k1

– Create third version of k1, first version of k2

– TTL for k1 is not updated (2591672), k2 gets the fresh ttl (2591994) from namespace default.

aql> EXECUTE c_or_u.create_or_update_record(2,4) ON test.demo where pk='k1'  
[
  {
    "digest": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=",
    "ttl": 0,
    "gen": 0,
    "bins": {
      "udf-fn:result": {
        "create_or_update_record": 0
      }
    }
  }
]
 
aql> EXECUTE c_or_u.create_or_update_record(2,4) ON test.demo where pk='k2'
[
  {
    "digest": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=",
    "ttl": 0,
    "gen": 0,
    "bins": {
      "udf-fn:result": {
        "create_or_update_record": 0
      }
    }
  }
 ]
 
aql> select * from test.demo
[
  {
    "digest": "qTPuDo+CwQZpcVDRXlBPCLKy4B0=",
    "ttl": 2591994,
    "gen": 1,
    "bins": {
      "bin1": 2,
      "bin2": 4
    }
  },
  {
    "digest": "t0f1hU0LMyWZKNDPq3+tgdar+/Y=",
    "ttl": 2591672,
    "gen": 3,
    "bins": {
      "bin1": 2,
      "bin2": 4
    }
  }
]

#14

@Albot In your function, you could move the bin assignments before and outside the if conditional loop. but for this topic, we can do:

function create_or_update_record (rec, k)
  if( not aerospike:exists( rec ) ) then
   rec['bin1']= 1   
   return aerospike:create(rec);
 else
   rec['bin1']= k   
   record.set_ttl(rec, -2)
   return aerospike:update(rec);
 end
end

Note: If I invoke this UDF as a background scan on all the records in namespace and set, I don’t think create() will be called. There is no key to call it on then … so it will only do the update() when applicable.

aql> EXECUTE c_or_u.create_or_update_record(2,4) ON test.demo


#16

Thanks a lot Team for support. UDF will serve this purpose. In my code, you can see that I am using increment and get using operate function. Is this still possible using UDF? If I get and return the value immediately after update will it work assuming UDF is atomic for that record.


#17

You cannot return a “record” type from a UDF. However you can return a bin value or a list of values of multiple bins or a map type key-value pairs from a UDF.

All ops inside the UDF are within the same record lock - so atomic. This is the benefit of using UDFs. However, if this is high throughput need, you must test and ensure you are getting the desired performance.

You have a unique implementation that even though it is a mutli-i/o operation, I could not see a scenario where you could be inconsistent. Even if your record expired between the CREATE_ONLY and UPDATE_ONLY calls and you miss the UPDATE_ONLY also, it would not matter because it would be close to expiration regardless after the UPDATE_ONLY had it succeeded. In other cases, you can use Generation policy equal to do a read-modify-write or Check-And-Set with optimistic locking.

So, in your use case, your code or the UDF method, both are viable - test and see which gives you better performance.

BTW, care to share what is your use case? ie what are you trying to achieve with this kind of code?


#18

Hi Gupta, Thanks for your detailed explanation. I have not benchmarked the UDF implementation. Will do it shortly.
My primary query in using UDF is, is it safe to use in production having 20mn hits an hour? Apart from this, below is my usecase for my code snippet.
I will be using this for frequency capping. Within a given frame of time, I will be able to push events only upto predefined number of times. Sending events more than once will be less than 10% of the total hits. Hence given preference for insert instead of update.

Whenever it goes for update, I see the latency is doubled as it goes for put, and on exception I am doing the Operate. It would be good if there is a policy to not touch the ttl if record already exists which saves one round trip.

Thanks for your support.


#19

If you are using for frequency capping and don’t need your data to persist, data type is integer, use single-bin record namespace with data-in-memory and data-in-index. You can store just these frequency capping implementation records in this namespace. Rest of your other persistent data you can put in a second namespace.

Your write rate - 20M /hour is about 5K per second which should be very doable - assuming you have adequate end to end hardware and network.

Re TTL -2 as a record exists action policy option, you can submit a feature request to Aerospike. https://discuss.aerospike.com/c/feature-requests


#20

This is a perfect reply. Yes I have already planned for single-bin integer config, you can observe from my code which has setname and binname as null.

Thanks a ton.


#21

– thats what happens when I reply at 2:00 am in the night. We already discussed that earlier in the thread. :slight_smile: