Partial writes to Aerospike Record when using Multiple Ops on a Record [Go Client]

Context: We have Go application using Aerospike for storage. In one of the use case, we perform Multiple Ops on a Record to perform multiple writes on a single key in one batch request. This is in production since a few months now and working fine.

Recently we found an issue related to this. While writing 4 bins values using func (clnt *Client) Operate(policy *WritePolicy, key *Key, operations ...*Operation) (*Record, error), we found that only one bin got stored in the record and no values for other three bins.

Code Sample:

    		key, err := as.NewKey("some_namespace", "unit", uniqueId)
		if err != nil {
			return err
		}

		typeValue := as.NewBin("type", type)
		docType := as.NewBin("doc_type", "m")
		updatedAt := as.NewBin("updated_at", fmt.Sprintf("%d", time.Now().UTC().Unix()))

		writePolicy := as.NewWritePolicy(0, 0)
		opTimeout := 5
		writePolicy.TotalTimeout = time.Duration(opTimeout) * time.Second
		writePolicy.SocketTimeout = time.Duration(opTimeout-1) * time.Second

		cdtListPolicy := as.NewListPolicy(as.ListOrderOrdered, as.ListWriteFlagsAddUnique)
		_, err = asClient.Operate(writePolicy, key,
			as.ListAppendWithPolicyOp(cdtListPolicy, departmentBin, departmentName),
			as.PutOp(typeValue),
			as.PutOp(docType),
			as.PutOp(updatedAt),
		)

Among all 4 bins (one list and other strings) only updatedAt value got stored in the record. Surprisingly, when we checked, the record has a generation value of 1, which means the record was not updated after insert. Also out of 4 bins, 2 are hardcoded in source code. And same code is being used since initial days

Multiple Ops on a Record document claims that multiple write operations will be in a transaction and will be atomic in nature. But then, we are not able to understand the cause of the above-mentioned case.

Please help us understand, how atomicity is managed in multiple operations and any chances of failure to ensure atomicity.

Aerospike Version: 5.1.0.10 build_os : ubuntu16.04 Edition: Enterprise Go Client Version: github.com/aerospike/aerospike-client-go, v2.9.0+incompatible

Thanks in advance.

If you look at the client’s test cases, there are a lot of them with multiple write operations. Does this also happen in your test cases, or is it a production issue only?

Thanks for your quick response.

It has happened in production only. Same code is in production for almost 5 months now. We never seen this issue before, this happened first time and in production only. Also, this happened only for very few records (< 10) and never reoccurred since then.

This is why we are not able to understand the use case.

Thanks

From your description, this does not seem to be a client issue. I’ll refer this issue to our server team to see if they have any immediate feedback.

1 Like

Can you share your server config file? Your settings may be relevant to the issue. Also, have you upgraded the server in the past few months?

Cluster Info: Nodes: 3 GCP Machine types: n1-standard-8

Config File: aerospike_conf.gz (1.3 KB)

Yes, we have upgraded the server from 4.8.0.8 to 5.1.0.10 around one month back.

@Khosrow_Afroozeh Do we found anything around this?

We’ve tried reproducing the issue without luck. Also didn’t find anything in a code review.

Are you seeing anything else when this happens? Such as warnings in the logs, increased latency, proxies, or anything else.

As of now, we haven’t received any other reports of the issue you’ve described.

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.