Server errors after system disk full

1

Hi, Configuration is the same as I posted before in thread Aerospike version is 3.3.12

We’re out of space on the system drive. Aerospike output to log:

Sep 14 2014 08:11:14 GMT: WARNING (rw): (thr_rw.c::2168) {ssd}: write_local_pickled: drives full Sep 14 2014 08:11:14 GMT: INFO (rw): (thr_rw.c::2523) [NOTICE] writing pickled failed(-1)::0x61605a0e804ad368793cb0abb751d52656983580 Sep 14 2014 08:11:14 GMT: WARNING (rw): (thr_rw.c::2927) {ssd}: write_local: drives full

After cleaning, aerospike continued write to log some message. df show disk full. I rotate log and run “kill -HUP cat /var/run/aerospike/asd.pid”

df show disk is free

But aerspike continued write som message and log grows very quickly.

AS restart helps, but it’s not a very good solution. Is it possible to solve such problems without restarting the Aerospike Server?

Thanks

2

It is possible that the Aerospike server is configured to use more storage than the system owns. The goal is to specify appropriately the amount of disk resource to be usable by Aerospike, in aerospike.conf. This allows Aerospike to manage storage via eviction and stop-write methods.

Here are some helpful steps to follow: http://www.aerospike.com/docs/operations/configure/namespace/storage/

Please also send your current aerospike.conf set up, and snap shot of “df”. We can also go through the configuration.

3

You can also run

sudo asmonitor -e collectinfo

which would gather debug info as outlined here:

http://www.aerospike.com/docs/tools/asmonitor/collectinfo.html

4

This error has been related to the problem description in Unused data on the disk - Error Code 8: Server memory error