Hello,
I just tried to install Aerospike on my VServer running Ubuntu 16.04.
After following the installation guide, the daemon does not start, and the following error occurs:
FAILED ASSERTION (hardware): (hardware.c:422) error while reading list of online CPUs
I googled a lot but cannot find any solution. From what I’ve learned Aerospike tries to check how many CPUs are online, however, that info is only available if the system supports hotplugging of CPUs. Which mine apparently doesn’t, since the file it looks for (/sys/devices/system/cpu/online) does not exist.
Does it require Hotplugging support? If yes, why? If no, how can I work around this? As I said before, I tried google but found nothing helpful.
Thanks for reaching out and for reporting this. Interesting. This is the first time that I see a kernel that wasn’t configured with CPU_HOTPLUG enabled. Can you do the following real quick on your system and send the output? I’m curious to learn more about the kernel you’re running.
dpkg --list | grep linux-image
uname -a
Yes, Aerospike does support hot-plugging or starting your kernel with the maxcpus=X option. That’s why it checks /sys/devices/system/cpu/online. However, Aerospike should probably just assume that all CPUs are actually online, when this entry is missing under /sys. It shouldn’t insist on this entry being there.
It’s just unusual to see a kernel that doesn’t have hot-plugging compiled in.
Before we look into a fix, let’s see what else is missing from /sys. Could you please also provide the output of the following?
sudo find /sys
I’m curious to see what /sys looks like with your kernel. Maybe there’s more that we need to fix.
My VServer is a V-Server Linux V40 from strato.de.
I actually already settled on another database system because I was urgently looking for an in-memory database and after not getting a reply for one day and being unable to fix it on my own I figured I’d move on, but if there’s more that I can do to help fix it I’ll do what I can.
Thank you very much for the additional information. I’m sorry to hear that you had to move on to a different database.
From the kernel version (4.4.0-042stab127.2) in the uname output it seems that Strato configure and build their own Linux kernels for use with their virtual servers, i.e., this isn’t a standard Ubuntu kernel. It looks like their kernel is pretty stripped down. They didn’t only remove the hot-plugging support, but a lot more: the /sys/devices/system/cpu/cpu0 directory is completely empty on your server, for example.
On startup, Aerospike gathers information about the CPU and memory configuration of the machine it runs on. This mechanism depends on the information provided by the kernel under /sys/devices/system/cpu. Unfortunately, all of this has been stripped away by Strato when they configured their custom kernel.
The Strato V40 is a para-virtualized machine (Virtuozzo). This means that the virtual servers (guests) share the Linux kernel of the underlying physical server (host). That’s why you didn’t find any packages named linux-image on your virtual server - your guest doesn’t have a kernel installed, because it uses the host’s kernel. Unfortunately, this also means that you cannot change the kernel, e.g., to a standard Ubuntu kernel. You’re stuck with the (host) kernel provided by Strato.
So, I’m afraid that it will be difficult to get Aerospike to work with a V40. I have opened an internal ticket, so that we’ll look into this, but the fix is non-trivial, I’m afraid. It would require a substantial rework of the performance optimization code that adapts Aerospike’s as well as the kernel’s configuration to the hardware it runs on, things like the number and topology of CPUs and NUMA nodes, NIC queues, etc.
AFAIK, there hasn’t been any customer requests for OpenVZ support. And very little on the community side as well. Have you requested support from OpenVZ on how to push the sysfs cpu topology information to the VM?