SSD storage-engine for Aerospike in Docker on a VPS not working

Operations Containers (Docker, etc.)
1

I am trying to run Aerospike in Docker with SSD as storage-engine device on a VPS. Following is my aerospike.conf file-

service {
	user root
	group root
	paxos-single-replica-limit 1
	pidfile /var/run/aerospike/asd.pid
	service-threads 4
	transaction-queues 4
	transaction-threads-per-queue 4
	proto-fd-max 15000
}

logging {
	file /var/log/aerospike/aerospike.log {
		context any warning
	}

	console {
		context any warning 
	}
}

network {
	service {
		address eth0
		port 3000
	}

	heartbeat {
		address eth0
		mode mesh
		port 3002
		interval 150
		timeout 10
	}

	fabric {
		address eth0
		port 3001
	}

	info {
		port 3003
	}
}

namespace test {
	replication-factor 1
	memory-size 2G
	default-ttl 5d

	storage-engine device {
		device /dev/vda1
		write-block-size 128K
		data-in-memory true
	}
}

The VPS has SSD and is mounted on /dev/vda1 as evident by the following-

root@test:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           798M  684K  797M   1% /run
/dev/vda1        49G   11G   38G  22% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda15      105M  3.6M  101M   4% /boot/efi
tmpfs           798M     0  798M   0% /run/user/0

However on deploying I get the following error-

link eth0 state up
link eth0 state up in 0
FAILED ASSERTION (drv_ssd): (drv_ssd.c:3194) unable to open device /dev/vda1: No such file or directory
WARNING (as): (signal.c:213) SIGUSR1 received, aborting Aerospike Community Edition build 4.6.0.2 os debian9
WARNING (as): (signal.c:215) stacktrace: registers: rax 0000000000000000 rbx 000000000000000a rcx 00007f193590bfbf rdx 0000000000000000 rsi 00007ffd023a55f0 rdi 0000000000000002 rbp 00007ffd023a5c20 rsp 00007ffd023a5668 r8 0000000000000000 r9 00007ffd023a55f0 r10 0000000000000008 r11 0000000000000246 r12 0000000000000011 r13 0000000000000000 r14 0000000000000080 r15 000055c2d04d3100 rip 00007f193590bfbf
WARNING (as): (signal.c:215) stacktrace: found 10 frames: 0x55c2ce7ceee7 0x7f193590c0e0 0x7f193590bfbf 0x55c2ce87546e 0x55c2ce85032c 0x55c2ce850cbb 0x55c2ce8535fd 0x55c2ce762d69 0x7f19346b92e1 0x55c2ce76374a offset 0x55c2ce713000
WARNING (as): (signal.c:215) stacktrace: frame 0: asd(as_sig_handle_usr1+0x11f) [0x55c2ce7ceee7]
WARNING (as): (signal.c:215) stacktrace: frame 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x7f193590c0e0]
WARNING (as): (signal.c:215) stacktrace: frame 2: /lib/x86_64-linux-gnu/libpthread.so.0(raise+0xcf) [0x7f193590bfbf]
WARNING (as): (signal.c:215) stacktrace: frame 3: asd(cf_fault_event+0x231) [0x55c2ce87546e]
WARNING (as): (signal.c:215) stacktrace: frame 4: asd(ssd_init_devices+0x410) [0x55c2ce85032c]
WARNING (as): (signal.c:215) stacktrace: frame 5: asd(as_storage_namespace_init_ssd+0x2c) [0x55c2ce850cbb]
WARNING (as): (signal.c:215) stacktrace: frame 6: asd(as_storage_init+0x34) [0x55c2ce8535fd]
WARNING (as): (signal.c:215) stacktrace: frame 7: asd(main+0x309) [0x55c2ce762d69]
WARNING (as): (signal.c:215) stacktrace: frame 8: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f19346b92e1]
WARNING (as): (signal.c:215) stacktrace: frame 9: asd(_start+0x2a) [0x55c2ce76374a]
FAILED ASSERTION (drv_ssd): (drv_ssd.c:3140) usable device size 0 must be greater than header size 8388608

I noticed that there is no /dev/vda1 inside the Aerospike container.

So to sum it up, how to use SSD storage-engine device for Aerospike in Docker on a VPS?

Reference-

1 Like
2

You probably don’t want to use vda1 since it is mounted as root (‘/’). Using vda1 would cause Aerospike to overwrite your root device. When using device the underlying device shouldn’t be mounted and Aerospike should be the only process writing to it.

To attach a raw device to a docker container, you need to use one of the --device options.

1 Like
3

I cannot have a separate raw SSD device to attach to Aerospike. Instead I want to use the SSD that comes with the VPS, but as you mentioned that it is not possible unless I use a SSD dedicated to Aerospike.

The only option I see is to use filesystem in storage-engine device and mount it as a docker volume. Am I correct?

4

Yes, volumes are the typical choice. Just in case it isn’t clear, you will need to use the file configuration rather than the device configuration.

5

Thank you for clarification-

storage-engine device {
	file /opt/aerospike/data/test.dat
	filesize 4G
	write-block-size 128K
	data-in-memory true
}

One last question, documentation says that for better performance on SSD write-block-size should be 128K, considering that I’m on a VPS with SSD storage and using file configuration should I use write-block-size 128K? I want to avoid OOM problem.

Also with the above file config I am getting the following log warning every 5 secs.

WARNING (hardware): (hardware.c:2262) failed to resolve mounted device /dev/vda1: 2 (No such file or directory)

I read this (High memory usage on Kubernetes GKE using helm chart) discussion but couldn’t find a solution.

6

The post-write-queue is a cache that holds recently written write-blocks. If you increase/decrease the size of the write-block-size you will increase/decrease the amount of RAM used by the post-write-queue. So it is possible to accidentally allocate all of your RAM to the post-write-queue if you aren’t careful.

7

Seems this is appearing several times today, please read this response: Issues with restarting an instance? - #6 by kporter

8

Thank you for your help so far. After this issue getting solved I’ve started running into OOM crash. Aerospike RAM consumption keeps increasing and crashes (OOM)

9

Saif, could you run the following two commands inside the container and share their output?

  • cat /proc/mounts
  • ls -l /dev

It does seem like /proc/mounts properly indicates that /opt/aerospike/data resides on /dev/vda1 . But it also seems like /dev does not contain a device node for vda1 . Hence I’d like to double-check the contents of /proc/mounts as well as what’s in /dev inside the container.

10 
$ cat /proc/mounts

overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/5GTHODYWF2QVOXZK4KIOM6BHAY:/var/lib/docker/overlay2/l/AB5AX524OTOUJZTTQ7PNFB6S4D:/var/lib/docker/overlay2/l/HK42LK4Y33U76QC47I7XJP56OT:/var/lib/docker/overlay2/l/SZXT6L4XGJ5PYEVGBLX7JPQ2UF:/var/lib/docker/overlay2/l/O2QKKT5J2PUQBKBHCGFVKNJ6JH,upperdir=/var/lib/docker/overlay2/1b509e29f9a2a204466179b34059627361cd49d1e744900eb7f4b86177e12fac/diff,workdir=/var/lib/docker/overlay2/1b509e29f9a2a204466179b34059627361cd49d1e744900eb7f4b86177e12fac/work 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,size=65536k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
cgroup /sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/pids cgroup ro,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/rdma cgroup ro,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/memory cgroup ro,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup ro,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
/dev/vda1 /etc/resolv.conf ext4 rw,relatime,data=ordered 0 0
/dev/vda1 /etc/hostname ext4 rw,relatime,data=ordered 0 0
/dev/vda1 /etc/hosts ext4 rw,relatime,data=ordered 0 0
/dev/vda1 /opt/aerospike/data ext4 rw,relatime,data=ordered 0 0
tmpfs /opt/aerospike/etc/aerospike.conf tmpfs ro,relatime 0 0
proc /proc/bus proc ro,relatime 0 0
proc /proc/fs proc ro,relatime 0 0
proc /proc/irq proc ro,relatime 0 0
proc /proc/sys proc ro,relatime 0 0
proc /proc/sysrq-trigger proc ro,relatime 0 0
tmpfs /proc/acpi tmpfs ro,relatime 0 0
tmpfs /proc/kcore tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/keys tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/timer_list tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/sched_debug tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/scsi tmpfs ro,relatime 0 0
tmpfs /sys/firmware tmpfs ro,relatime 0 0

$ ls -l /dev

total 0
lrwxrwxrwx 1 root root   11 Sep 26 12:08 core -> /proc/kcore
lrwxrwxrwx 1 root root   13 Sep 26 12:08 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 Sep 26 12:08 full
drwxrwxrwt 2 root root   40 Sep 26 12:08 mqueue
crw-rw-rw- 1 root root 1, 3 Sep 26 12:08 null
lrwxrwxrwx 1 root root    8 Sep 26 12:08 ptmx -> pts/ptmx
drwxr-xr-x 2 root root    0 Sep 26 12:08 pts
crw-rw-rw- 1 root root 1, 8 Sep 26 12:08 random
drwxrwxrwt 2 root root   40 Sep 26 12:08 shm
lrwxrwxrwx 1 root root   15 Sep 26 12:08 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root   15 Sep 26 12:08 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root   15 Sep 26 12:08 stdout -> /proc/self/fd/1
crw-rw-rw- 1 root root 5, 0 Sep 26 12:08 tty
crw-rw-rw- 1 root root 1, 9 Sep 26 12:08 urandom
crw-rw-rw- 1 root root 1, 5 Sep 26 12:08 zero
11

Thanks for your quick response, Saif! Yes, as suspected, the Docker container says that /dev/vda1 is mounted at /opt/aerospike/data. (And also, strangely, at resolv.conf, hostname, and hosts.) See the following excerpt from /proc/mounts.

/dev/vda1 /etc/resolv.conf ext4 rw,relatime,data=ordered 0 0
/dev/vda1 /etc/hostname ext4 rw,relatime,data=ordered 0 0
/dev/vda1 /etc/hosts ext4 rw,relatime,data=ordered 0 0
/dev/vda1 /opt/aerospike/data ext4 rw,relatime,data=ordered 0 0

Let’s leave resolv.conf, hostname, and hosts aside for now - I’ll have to research this a little more. It seems unusual to see a single device (a) mounted multiple times and (b) as a file instead of a directory.

So, let’s go back to the warning you’re seeing. As /proc/mounts says that it’s /dev/vda1 that’s mounted at /opt/aerospike/data, the Aerospike server tries to take a look at /dev/vda1. But vda1 does not exist in your /dev directory. So, Aerospike cannot find /dev/vda1 - hence the warning.

The warning is likely triggered periodically, because something periodically queries the cluster node’s SSD health information, which is part of the cluster node’s namespace stats. So, if you ran asinfo -v namespace/foobar against this node, then this would trigger the warning.

It’s safe to just ignore this warning for now. You won’t get any SSD health information, but this only works for physical NVMe drives anyway. So, you aren’t losing any functionality in your configuration.

My understanding of Docker’s /dev management is very limited, so I’ll have to figure out how to best avoid this warning. For now, however, my recommendation would be to ignore it.

1 Like
12

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.