Aerospike's Disk not getting mount after machine restart


#1

Hi

Came across this scary problem. Our box running Aerospike server rebooted. And disk which was allocated to AS couldn’t get mounted by itself. Appropriate entries of this disk were in fstab. Reason being linux couldn’t identify the file system.

We were banking on disk storage to for recovery purpose. But with this problem, that’s looking impossible now.

Can someone pls suggest a solution? Is there any known filesystem name we can provide to the mount command ?

Or should we rather switch to using filesystem rather than using raw drive ? Will there be any perf hits with file system compared to raw device ? KIndly help here. Thanks.


#2

I’m a bit confused. Are you writing to a raw device which is mounted? If so Aerospike will happily overwrite the filesystem for you :smile:. Be extra careful not to supply your root device, have personally made that mistake :grimacing:.

There is a significant performance hit for going through the filesystem, there are ways to reduce the hit by mounting with option such as noatime but you will not regain all of the lost performance.

I guess you should be able to continue using the disk? If not and you have multiple disks [used by Aerospike] on this node, you can zeroize this disk and then start the server, Aerospike will treat it as a disk replacement and migrations will repopulate the disk. If you have only one disk, this is still true but less interesting :smiley:.

EDIT: Added “used by Aerospike”


#3

Hello @kporter

Thanks for your reply. Please see my inline comments.

I’m a bit confused. Are you writing to a raw device which is mounted? If so Aerospike will happily overwrite the filesystem for you smile. Be extra careful not to supply your root device, have personally made that mistake grimacing.

Yes, we are writing on a mounted raw device, specially mounted for AS use. And yes, AS is happily overwriting the filesystem. And I should confess here that once in past I also ended up allocating the disk with OS installed to AS and it was rightfully wiped off(This is not the reason of my current problem though)

I guess you should be able to continue using the disk? If not and you have multiple disks on this node, you can zeroize this disk and then start the server, Aerospike will treat it as a disk replacement and migrations will repopulate the disk. If you have only one disk, this is still true but less interesting

We have multiple disks on system. One of them is allocated to AS. Other disks are in ext4 while AS has created its own file system. Everything was running fine till here. Now there was some hardware issue due to which the machine rebooted . All the other disks gets mounted by themselves except disk that was getting used by AS

Linux doesn’t doesn’t identify AS’s format and doesn’t mount that disk. It gives error: "mount: you must specify the filesystem type"

One has to wipe clean that disk and then mount it manually in a different step. And this is my problem actually. I want the AS disk to get mount itself (like others) and then do a cold restart when the machine reboots.

Can you please suggest a solution for it without actually using the file-system in place of raw device? Sorry if I am still not clear. Let me know what specific I can explain more. Thanks.


#4

When using raw devices, those devices shouldn’t be mounted.


#5

Always used to think that disks should be mounted to be used. :expressionless:

Sorry to bother you for my mistake. I will use it without mounting. Thanks again!