Unable to write to set

Hi,

Since early this week all our writes to a specific set has stopped completely. As a result, our system is severely hampered. All we see is error code -1. Our service is a high traffic system and this set was able to read & write until last week. But suddenly since 2-3 days ago, it’s stopped.

{“level”:50,“time”:1677094410152,“pid”:90,“hostname”:“us-wdc-lw03-11677415”,“error”:{“name”:“AerospikeError”,“code”:-1,“command”:{“client”:{“_events”:{},“_eventsCount”:0,“config”:{“port”:3000,“hosts”:[{“addr”:“172.17.0.47”,“port”:3000},{“addr”:“172.17.1.149”,“port”:3000},{“addr”:“172.17.2.89”,“port”:3000}],“policies”:{“write”:{“totalTimeout”:5000,“maxRetries”:0,“exists”:4},“read”:{“totalTimeout”:5000,“maxRetries”:0}},“log”:{“level”:-1,“file”:2},“connTimeoutMs”:10000,“sharedMemory”:{“key”:2768240640},“useAlternateAccessAddress”:false},“as_client”:{},“connected”:true,“captureStackTraces”:false},“args”:[{“ns”:“‘impressionsCache’”,“set”:“‘throttlingCountV2’”,“key”:“1|domain_centurylink.net|country_USA|device_Desktop|mediaType_Banner”,“digest”:{“type”:“Buffer”,“data”:[180,103,34,56,27,61,230,36,12,13,211,24,49,24,24,244,103,69,73,123]}},{“0379d14bb5499c9”:{“id”:1,“key”:“1|domain_centurylink.net|country_USA|device_Desktop|mediaType_Banner”,“createdAt”:1677094410,“bidsCount”:0,“bidRequestsCount”:1,“isThrottlingEnabled”:false}},{“ttl”:960},{“totalTimeout”:1000,“exists”:1}],“captureStackTraces”:false,“key”:{“ns”:“‘impressionsCache’”,“set”:“‘throttlingCountV2’”,“digest”:{“type”:“Buffer”,“data”:[180,103,34,56,27,61,230,36,12,13,211,24,49,24,24,244,103,69,73,123]}},“ensureConnected”:true},“func”:“as_partition_info_init”,“file”:“src/main/aerospike/as_partition.c”,“line”:318,“inDoubt”:false},“key”:“1|domain_centurylink.net|country_USA|device_Desktop|mediaType_Banner”,“message”:“Generic client API usage error.”,“exchangeError”:true,“msg”:“Error while storing data in aerospike cache”}

If someone can point us in the right direction regarding this error, it would be greatly appreciated.

-1 looks like its some generic catch-all I guess. What version of server and client are you using? What troubleshooting have you done? Have you looked through the aerospike logs for any clues? grep -v INFO aerospike.log is a good easy server check, or journalctl -xu aerospike --since=-1d | grep -v INFO|less if its journaled maybe (we use logfile so that might not be 100% right)

Are you able to reproduce this is any cluster/set? What kind of operation is it, is there some CDT or Expression present? Can you share a small sample to reproduce this is some env? Does asadm -e 'health -n 3 -v' show anything interesting? Whats the output of asinfo -lv 'sets' ?

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.