We recently deployed our ruby application with aerospike on a test server with production like load and within a matter of few seconds we started getting this error very frequently. We were doing a single get from aerospike in each request with the user_id as key. The value for each key is a list with upto two small hashes as its elements which in no case can be greater than the
MAX_BUFFER_SIZE which is
10MB by default. There is no other call to the aerospike.
We are using nginx passenger setup with passenger running 50 app instances. In the
config/initializers we’ve initialized the aerospike client as a global variable (
$as_client). A little digging in the code shows the error comes while parsing the headers from the
data_buffer. Is it possible that the data_buffer gets corrupted by multiple requesting using it simultaneously? The error message in the logs is like :
Aerospike::Exceptions::Parse: Invalid size for buffer: 1602040889240
The size mentioned in the message is always way bigger than the actual size of the data in that key. How do I go about debugging this? I tried creating a new client instance for every request and it seemed to work fine without giving any errors. However, when I have only one client instance initialized in the initializers direcrory as a global variable, this error comes up.