One node disk space suddenly free falls to 0% free disk


#1

We’ve been using using both enterprise and community versions of aerospike (right now 3.7.4.1, this happened with previous versions)

Our cluster size has been 32 and 8 (different instance class) in AWS

Suddenly one nodes disk free will start free-falling until it gets to 0. Currently I erase the data and start it up again

Only thing I see in the logs is:

Mar 22 2016 04:22:11 GMT: WARNING (drv_ssd): (drv_ssd.c::4163) {user_profiles} write fail: queue too deep: q 68, max 64
Mar 22 2016 04:22:11 GMT: WARNING (drv_ssd): (drv_ssd.c::4163) {user_profiles} write fail: queue too deep: q 68, max 64
Mar 22 2016 04:22:11 GMT: WARNING (drv_ssd): (drv_ssd.c::4163) {user_profiles} write fail: queue too deep: q 67, max 64
Mar 22 2016 04:22:11 GMT: WARNING (drv_ssd): (drv_ssd.c::4163) {user_profiles} write fail: queue too deep: q 67, max 64

Any ideas on this?

Thanks


#2

Queue too deep indicates that you are writing faster than your disk[s] can process.