Handling nulls in maps in expressions

smarkes · May 21, 2024, 4:48pm

I’ve got a problem handling maps with potential null values in expressions. Let’s say I have a bin

"binName": {
    "key0": null,
    "key1": "value",
}

I want to create an expression which will check if key already exists and either create it with new value or update it only when existing one is null. Unfortunately if the existing value is null I’m getting either AEROSPIKE_ERR_REQUEST_INVALID or AEROSPIKE_ERR_OP_NOT_APPLICABLE

After some testing with read operations using expression, the only way I can even get null value is by using

exp.MapGetByKey(None, aerospike.MAP_RETURN_KEY_VALUE, exp.ResultType.MAP, 'key0', exp.MapBin('binName'))

Eg. using aerospike.MAP_RETURN_VALUE with exp.ResultType.STRING will result in AEROSPIKE_ERR_OP_NOT_APPLICABLE for keys that hold null value. Using exp.ResultType.MAP would be fine if I could somehow use this result in an Eq expression. But I can’t figure it out.

pgupta · May 21, 2024, 6:26pm

I have modeled your problem in Jupyter Notebook using interactive Java - so I will share each relevant cell with you. It should give you the pertinent implementation you are looking for.

I started with a server with no records / clean / namespace truncated.

First, I will add 4 records with a nested map as a general example.

void addRecord(Integer keyIndex, String mapBinName, String mapKey1Val, String mapKey2Val, String subMapKey1Val, boolean addKey1){
    MapPolicy mPolicy = new MapPolicy(MapOrder.UNORDERED, MapWriteFlags.DEFAULT);
    WritePolicy wPolicy = new WritePolicy();
    wPolicy.sendKey = true;  //Optional, if you want to inspect the record key
    Key myRecKey = new Key("test", "testset", Value.get("key"+keyIndex));
    HashMap <String, Value> profileObj = new HashMap <String, Value>();
    if(addKey1) { profileObj.put("Key1", Value.get(mapKey1Val));  } 
    profileObj.put("Key2", Value.get(mapKey2Val));  
    
    //Sub-map (optional)
    HashMap <String, String> metaObj = new HashMap <String, String>();
    metaObj.put("Key1", subMapKey1Val);
    profileObj.put("subKey1", new MapValue(metaObj)); 
    

    client.operate(wPolicy, myRecKey, 
       MapOperation.put(mPolicy, mapBinName, Value.get("profile"), new MapValue(profileObj) )               
    );
    System.out.println("Rec added: [key "+keyIndex+"]: "+ client.get(null, myRecKey));
}

Insert the test records:

//Create test records
String binName = "myMapBin1";
addRecord(1, binName, "val11", "val12", "vals11", true);
addRecord(2, binName, "val21", "val22", "vals21", true);
addRecord(3, binName, "val31", "val32", "vals31", false);
addRecord(4, binName, null, "val42", "vals41", true);

Output: (Note: key3 and key4 have the two cases for nested key Key1 that you are interested in. (I added a nesting level just to show use of CTX as well.)

Rec added: [key 1]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val12, Key1=val11, subKey1={Key1=vals11}}}))
Rec added: [key 2]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val22, Key1=val21, subKey1={Key1=vals21}}}))
Rec added: [key 3]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val32, subKey1={Key1=vals31}}}))
Rec added: [key 4]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val42, Key1=null, subKey1={Key1=vals41}}}))

Now, I will update all the records but expecting only case key3 and key4 to update. The updated key value is: val31_updated and val41_updated etc.

Code to do the update. (This filter expression is I believe what you are looking for.)

void updateRecord(Integer keyIndex, String mapBinName, String mapKey1Val){
    MapPolicy mPolicy = new MapPolicy(MapOrder.UNORDERED, MapWriteFlags.DEFAULT);
    WritePolicy wPolicy = new WritePolicy();
    wPolicy.sendKey = true;  //Optional, if you want to inspect the record key
    Key myRecKey = new Key("test", "testset", Value.get("key"+keyIndex));
    
    Expression mapKeyFilter = Exp.build(
        Exp.eq(
            MapExp.getByKey(MapReturnType.VALUE, Exp.Type.NIL, Exp.val("Key1"), Exp.mapBin(mapBinName), CTX.mapKey(Value.get("profile"))),
            Exp.nil() )
    );
 
    wPolicy.filterExp = mapKeyFilter;
    
    client.operate(wPolicy, myRecKey, 
       MapOperation.put(mPolicy, mapBinName, Value.get("Key1"), Value.get(mapKey1Val), CTX.mapKey(Value.get("profile")) )              
    );
    System.out.println("Read record: [key "+keyIndex+"]: "+ client.get(null, myRecKey));
}

Run the update on all records, expect only Key1 missing or value null should update:

//Update all records, should only update ones with Key1 missing or Key1 value null
String binName = "myMapBin1";
updateRecord(1, binName, "val11_updated");
updateRecord(2, binName, "val21_updated");
updateRecord(3, binName, "val31_updated");
updateRecord(4, binName, "val41_updated");

Output:

Read record: [key 1]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val12, Key1=val11, subKey1={Key1=vals11}}}))
Read record: [key 2]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val22, Key1=val21, subKey1={Key1=vals21}}}))
Read record: [key 3]: (gen:2),(exp:0),(bins:(myMapBin1:{profile={Key2=val32, Key1=val31_updated, subKey1={Key1=vals31}}}))
Read record: [key 4]: (gen:2),(exp:0),(bins:(myMapBin1:{profile={Key2=val42, Key1=val41_updated, subKey1={Key1=vals41}}}))

smarkes · May 21, 2024, 10:07pm

@pgupta Thank you. I checked it quickly and using filter expression would work. But there is a big problem with it. This would require separate operate call per key I want to update for each record. But there are other operations made on the record’s other bins and potentially lots of key updates in this one bin. The write expression looked perfect for the job and if not for the nulls it would actually work.

I would rather avoid UDF if possible.

pgupta · May 22, 2024, 12:20am

Here is a different version of updateRecord() that may do what you want. It uses ExpOperation.write to update conditionally and with ExpWriteFlag.EVAL_NO_FAIL so for records that don’t match the condition, you don’t generate an exception.

void updateRecord(Integer keyIndex, String mapBinName, String mapKey1Val, String mapKey2Val){
    MapPolicy mPolicy = new MapPolicy(MapOrder.UNORDERED, MapWriteFlags.DEFAULT);
    WritePolicy wPolicy = new WritePolicy();
    
    wPolicy.sendKey = true;  //Optional, if you want to inspect the record key
    Key myRecKey = new Key("test", "testset", Value.get("key"+keyIndex));

    Expression key1ValExp = Exp.build( 
        Exp.cond( 
            Exp.eq(
              MapExp.getByKey(MapReturnType.VALUE, Exp.Type.NIL, Exp.val("Key1"), Exp.mapBin(mapBinName), CTX.mapKey(Value.get("profile"))),
              Exp.nil()
            ), 
            MapExp.put(mPolicy, Exp.val("Key1"), Exp.val(mapKey1Val), Exp.mapBin(mapBinName), CTX.mapKey(Value.get("profile"))),            
            Exp.unknown()        
        )
    );

    client.operate(wPolicy, myRecKey,        
        ExpOperation.write(mapBinName, key1ValExp, ExpWriteFlags.EVAL_NO_FAIL),
        MapOperation.put(mPolicy, mapBinName, Value.get("Key2"), Value.get(mapKey2Val), CTX.mapKey(Value.get("profile")))
    );
    System.out.println("Read record: [key "+keyIndex+"]: "+ client.get(null, myRecKey));
}

Starting with the same set of records as before:

Rec added: [key 1]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val12, Key1=val11, subKey1={Key1=vals11}}}))
Rec added: [key 2]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val22, Key1=val21, subKey1={Key1=vals21}}}))
Rec added: [key 3]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val32, subKey1={Key1=vals31}}}))
Rec added: [key 4]: (gen:1),(exp:0),(bins:(myMapBin1:{profile={Key2=val42, Key1=null, subKey1={Key1=vals41}}}))

I am modifying {profile:{ Key2: val} ...} for all the 4 records, updating Key1 only for the two records key3 (Key1 absent) and key4 that has value for Key1 as null.

Run:

String binName = "myMapBin1";
updateRecord(1, binName, "val11_updated", "val12_updated");
updateRecord(2, binName, "val21_updated", "val22_updated");
updateRecord(3, binName, "val31_updated", "val32_updated");
updateRecord(4, binName, "val41_updated", "val42_updated");

Output:

Read record: [key 1]: (gen:2),(exp:0),(bins:(myMapBin1:{profile={Key2=val12_updated, Key1=val11, subKey1={Key1=vals11}}}))
Read record: [key 2]: (gen:2),(exp:0),(bins:(myMapBin1:{profile={Key2=val22_updated, Key1=val21, subKey1={Key1=vals21}}}))
Read record: [key 3]: (gen:2),(exp:0),(bins:(myMapBin1:{profile={Key2=val32_updated, Key1=val31_updated, subKey1={Key1=vals31}}}))
Read record: [key 4]: (gen:2),(exp:0),(bins:(myMapBin1:{profile={Key2=val42_updated, Key1=val41_updated, subKey1={Key1=vals41}}}))

For records key1 and key2 that already have values val11 and val21 in their Key1, those are not updated to xx_updated.

smarkes · May 22, 2024, 8:02am

@pgupta Oh, this actually works. The reason I couldn’t find it before is that python doesn’t define a constant for result type null. So I didn’t try to experiment with it before. But after your java example I just tried to run those expressions with result type hardcoded to 0 and it works.

So I think I can manage from here. Thanks a lot!

skruczek · May 28, 2024, 8:36pm

Hi @smarkes I am just reaching out to see if you got all of your questions answered and appreciate your contribution here. As an FYI, we also have a Discord community where you can talk to our DevRel team directly as well. Always helpful to get feedback and let us know if you need any more info. Let me know if you have any other questions. Cheers! Stacey K, DevRel Team lead

system · August 20, 2024, 8:37pm

This topic was automatically closed 84 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Context within expressions not working as expected Python Client	3	921	November 10, 2021
Filter map by key-value pairs Go Client expressions	3	1423	September 9, 2021
How to make expression evaluate to nil in order to delete bin Python Client	6	863	November 16, 2021
Writing `aerospike.null()` to map doesn't remove item from map Python Client	2	824	April 25, 2022
Issue with aerospike ordered_map map	5	1466	August 21, 2018

Handling nulls in maps in expressions

Related topics