Updating a nested map fails

Hello community experts!

I’ve stumbled upon the following use case which involves nested Map operations: My record contains a bin of Map data type. I need to find a value (a nested Map) inside the top-level map and in case the nested Map found I need to update a field in it (created value) and return the whole nested map value to a client. If I call a “put” operation on a nested map for a not-existing record the operation fails. The same “put” operation called on a top-level map for a not-existing record succeeds.

Here’s my bin structure:

bin = { key1 = { created=1, value=value1 }, key2 = { created=2, value=value2 } }

I need to read a nested map for key = key1 and update its key = “created”. Here’s my code:

public class NestedMapUpdate {

    public static void main(String[] args) {
        try {
            AerospikeClient client = new AerospikeClient("localhost", 3000);
            Key recordKey = new Key("test", "test", "record1");

            Value key1 = Value.get("key1");

            MapPolicy policy = new MapPolicy(MapOrder.UNORDERED, MapWriteFlags.UPDATE_ONLY | MapWriteFlags.NO_FAIL);

            Operation[] readOps = {
                MapOperation.put(policy, "map", Value.get("created"), Value.get(123L), CTX.mapKey(key1)),
                MapOperation.getByKey("map", key1, MapReturnType.VALUE)
            };
            client.operate(client.writePolicyDefault, recordKey, readOps);
        } catch (AerospikeException e) {

            e.printStackTrace(System.err);
        }
    }
}

the following exception is thrown:

com.aerospike.client.AerospikeException: Error 4,1,30000,0,0,BB9B135BB0E1B90 88.99.69.12 3000: Parameter error
    at com.aerospike.client.command.ReadCommand.parseResult(ReadCommand.java:173)
    at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:97)
    at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:50)
    at com.aerospike.client.AerospikeClient.operate(AerospikeClient.java:1288)
    at ru.cleverdata.dmpkit.idgraph.aerospike.AerospikeIssue.main(AerospikeIssue.java:32)

As you can see I passed MapWriteFlags to put operation which contains NO_FAIL bits set. So I expected the nested operation will do nothing in case the top-level map or the whole record isn’t found.

How do I solve my case?

aerospike-server: 4.6.0.4 aerospike-client: 4.4.7

Another related question is when I do the same put operation on the top-level map for a non existing record the record gets created with a bin which value is null. MapPolicy defined as

MapWriteFlags.UPDATE_ONLY | MapWriteFlags.NO_FAIL

I expect that no record is created based on the specified MapWriteFlags value.

The code snippet which reproduces this behaviour:

public class NestedMapUpdate {

    public static void main(String[] args) {
        try {
            AerospikeClient client = new AerospikeClient("dmpkit-dev-mn2", 3000);
            Key recordKey = new Key("test", "test", "record4");
            Value mapKey = Value.get("key1");

            MapPolicy policy = new MapPolicy(MapOrder.UNORDERED,
                MapWriteFlags.UPDATE_ONLY | MapWriteFlags.NO_FAIL);

            Operation[] ops = {
                MapOperation.put(policy, "map", mapKey, Value.get(123L)),
            };
            client.operate(client.writePolicyDefault, recordKey, ops);

            // the following record is created despite of MapPolicy
            Record record = client.get(client.readPolicyDefault, recordKey);

        } catch (AerospikeException e) {
            e.printStackTrace(System.err);
        }
    }
}

Thank you very much in advance!

1 Like

The no_fail policy means that the server will not fail on policy violations.

There is a feature being planned to allow a write operation to create the context similar to mkdir -p. Maybe no_ fail will apply then, since that change makes this failure a policy decision.

Hi Kevin! Thank you so much for your reply!

Does the flag MapWriteFlags.UPDATE_ONLY | MapWriteFlags.NO_FAIL mean that if there is no record in the set there’ll be no insert introduced and no exception thrown? But in reality it seems not to be working the way you described, as when I call “put” operation with the policy defined above:

client.operate(
	client.writePolicyDefault, 
	recordKey, 
	new Operation[]{ MapOperation.put(policy, "map", mapKey, Value.get(123L))}
);

an exception is thrown:

com.aerospike.client.AerospikeException: Error 4,1,30000,0,0,BB9B135BB0E1B90 88.99.69.12 3000: Parameter error

Could you please confirm if it is an expected behaviour for a nested operation? In fact, though the exception is thrown despite of a no_fail flag.

Kind regards, Mike.

This isn’t a nested operation, I’m not sure why this would fail. When this failure occurs, the aerospike.log should contain a warning containing more information about that failure. Could you provide the output of that warning?

My expectation here is that the record doesn’t exits, so we will attempt to create a record (assuming your writePolicyDefault allows creation). Since the record didn’t exist the map will also not exist and the put will fail. Because this is a policy violation, that failure will be replaced with success.

Kevin is correct in that NO_FAIL only applies to the context it is in and has nothing to do with the existence of the parent context. If the parent context does not exist, the op will still fail.

This later part is addressing your 2nd post. I did a test and MapOperation.put() doesn’t fail if you have UPDATE_ONLY | NO_FAIL. The record does get created in this case though. But no exception is thrown as you described.

Output:

        generation 1, ttl 2592000, 1 bin:
          testmap : 0
        get and print
          generation 1, ttl 2592000, 1 bin:
          testmap : {}

Test Code:

	as_key rkey;
	as_key_init_int64(&rkey, NAMESPACE, SET, 2);

	as_record* rec = NULL;
	as_error err;
	as_status status = aerospike_key_remove(as, &err, NULL, &rkey);
	assert_true(err.code == AEROSPIKE_OK || err.code == AEROSPIKE_ERR_RECORD_NOT_FOUND);

	as_operations ops;
	as_arraylist empty_list;
	as_arraylist_init(&empty_list, 1, 1);

	as_map_policy pol;
	as_map_policy_set_flags(&pol, AS_MAP_UNORDERED, AS_MAP_WRITE_UPDATE_ONLY | AS_MAP_WRITE_NO_FAIL);

	as_operations_init(&ops, 1);
	as_operations_add_map_put(&ops, BIN_NAME, &pol, (as_val*)as_string_new_strdup("sfcd"), (as_val*)&empty_list);

	status = aerospike_key_operate(as, &err, NULL, &rkey, &ops, &rec);
	assert_int_eq(status, AEROSPIKE_OK);
	as_operations_destroy(&ops);
	example_dump_record(rec);
	as_record_destroy(rec);
	rec = NULL;

	info("get and print")
	status = aerospike_key_get(as, &err, NULL, &rkey, &rec);
	assert_int_eq(status, AEROSPIKE_OK);
	dump_record(rec);
	as_record_destroy(rec);
	rec = NULL;

Thanks a lot for all your replies!

I’m really sorry for copying the wrong line from the adjacent code sample. It should’ve been the line with a nested operation, here is the correct one:

client.operate(
	client.writePolicyDefault, 
	recordKey,
	MapOperation.put(policy, "map", Value.get("created"), Value.get(123L), CTX.mapKey(mapKey))
);

Here it is:

Dec 18 2019 06:52:11 GMT: WARNING (particle): (cdt.c:1116) cdt_process_state_context_eval() bin type 0 is not list or map
Dec 18 2019 06:52:11 GMT: WARNING (rw): (write.c:2034) {test} write_master: failed as_bin_cdt_stack_modify_from_client() <Digest>:0xfd6fbae7ea1b4b8256f996cb29d89fb20e7aa054

This is exactly what I expected too, but unfortunately I faced a failure.

That’s true, but doesn’t it contradict the update_only flag? Thank you so much for implementing a test! I need to clarify things a bit, sorry for confusion.

I’ve come come across two uncertainties:

  1. Nested MapOperation.put operation always fails when no record exists beforehand, ignoring no_failure flag.
  2. Normal MapOperation.put (not a nested one!) respects no_failure flag, but ignores update_only flag and creates a record if one doesn’t exist before the operation call. Your test case proves exactly that.

I just need to clarify things as one of our business critical features depends on this behaviour. Let me know if you need the business feature in-depth description, I’ll provide it with pleasure.

Thank you all in advance.

All the best, Mike.

There’s two separate systems in play here.

  1. Nested is a wholly CDT system, we do not yet support creating context on the fly, so you have to do it beforehand in another op.
  2. This is the database record system behavior. It can have behavior that is different from the CDT subsystem. When you do a modify op on a record that doesn’t exist, it creates a record for you. And the bin gets created too.

The UPDATE_ONLY/CREATE_ONLY flags only govern adding the key-value map element. It does not have jurisdiction over anything beyond that, such as creating a record or even the bin.

In this case, the key-value pair was not created, so it is consistent with the policy specified. And to re-iterate, NO_FAIL only checks violations of UPDATE_ONLY or CREATE_ONLY, and nothing else.

Thank you for the in-depth description. It’s all clear for me now,

Could you please clarify what you mean here by creating a context in another op? Let’s examine a real-life use case:

I need to read a nested map and update the nested map value in case it exists and do nothing if any of the following is true:

  • a record doesn’t exist
  • a bin doesn’t exist,
  • the bin doesn’t contain a nested map,
  • the nested map doesn’t contain a requested key.

The expected behaviour of a nested CDT looks exactly like Update_Only WritePolicy for the Record System. I don’t think creating a record with an empty bin is a way to go, because a sufficient amount of such requests will consume tones of RAM.

Thank you.

Just to sum everything up, the behavioural duality of the Record System and nested CDT boils down to implementation differences.

Speaking about the future, do you have any plans for unifying the behaviour of these systems?

There’s no plan to unify those systems. I doubt it will even be possible without breaking all sorts of behavior most people currently depend on. There is a plan however, to implement creating a sub context where they did not exist before so some things can be done in 1 op.