Error Message when trying to do writes (client.puts()) to our DB

Client Libraries Node.js Client
1

Hi guys

I’m getting the following error message when trying to do write’s (client.puts()) to our DB:

{ code: 1, message: ‘AEROSPIKE_ERR_SERVER’, func: ‘as_command_parse_header’, file: ‘src/main/aerospike/as_command.c’, line: 485 }

This isn’t occurring with any other function, such as queries or reads . I’ve checked other forum posts and haven’t managed to find a solution, I’ve changed timeout settings in the config as per one of the threads but to no effect.

OS: Ubuntu 12.04 LTS, Node Client Version: 1.0.39 (although was also failing on 1.0.35), Server is pulled from the latest EC2 AMI.

Thanks for any insight you can give.

John

2

Hi John,

AEROSPIKE_ERR_SERVER, is a generic error returned by the server, when the request is not a valid one. Could you give us more details about the nature of put requests and also could you look at the aerospike server log, when this error is returned?

Thanks

Gayathri.K

3

Hi Gayathri

Thanks for your reply.

To give some context, the namespace has been working for the last week with the same put request but suddenly it started to reply with the 500 error (at around the 60mill master object level). When we tried to perform other functions such as reading or putting with an existing key everything was fine, only when adding a new object did we receive the error. Below is the warning messages from the logs we we’re getting:

May 22 2015 17:28:50 GMT: WARNING (cf:arenax): (arenax_cold.c::45) could not allocate 1073741824-byte arena stage 4

May 22 2015 17:28:50 GMT: WARNING (index): (index.c::241) arenax alloc failed

May 22 2015 17:28:50 GMT: WARNING (rw): (thr_rw.c::3546) {newbids} write_local: fail as_record_get_create() :0x92988abfa81220eb3eb13cd63ccd7d2990b01bdc

May 22 2015 17:28:50 GMT: INFO (drv_ssd): (drv_ssd.c::2398) device /dev/xvdg: used 48385680768, contig-free 363323M (2906591 wblocks), swb-free 2, n-w 0, w-q 0 w-tot 371139 (0.0/s), defrag-q 0 defrag-tot 595 (0.0/s)

I’ve also included that last info to show it’s not a space issue. The issue has evolved further where aerospike will restart without warning but will only work when the namespace is removed from the config.

It feels like a possible namespace corruption but I’m not sure how to check in more detail.

Thanks for your help

4

Hi John,

The system is failing to allocate the memory required to create index for the record. Indices are stored in RAM, and the memory is allocated in chunks of 1GB. For the writes after 60 million objects, the system tries to allocate memory for index and that is failing. This is what we inferred from the log file.

The storage device has enough space to store the record as per your log.

Could you give us the machine configuration on which you are running Aerospike Server and also the approximate number and size of objects you would be storing in Aerospike ?

Thanks

1 Like
5

Hey Gayathri

You’ve solved it :smile:

While I’d provisioned for data storage I forgot about the index RAM requirement and used the majority RAM space for another namespace.

Thanks for your assistance with this

John

1 Like