Kubernetes failed to resolve mounted device devdsk1 2 No such file or directory

Knowledge Base Deploying
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Kubernetes: failed to resolve mounted device /dev/dsk1: 2 (No such file or directory)

Problem Description

In a Kubernetes based environment where a file based storage device is being used, the following message is repeated frequently in the aerospike.log file:

Feb 20 2020 16:22:49 GMT: WARNING (hardware): (hardware.c:2258) failed to resolve mounted device /dev/dsk1: 2 (No such file or directory)

Explanation

When a file based storage-engine is in use the Aerospike hardware module code will try and look up the parent device for the filesystem in use. The following scenarios can trigger this check to be executed.

  • Checking namespace stats via an info call e.g.
    • storage device stats
    • index device stats (all-flash which has mount points) or pmem mount points (can occur anytime the “namespace/” info command is called)
  • Once during Aerospike daemon (asd) startup if using pmem devices.
  • Setting scheduler for storage devices.

Most commonly, the check would be triggered by checking namespace statistics when populating the metrics for device age or lifetime. These are:

Specifically, the warning is caused in a Docker or Kubernetes environment as the parent device (such as /dev/sda) is not visible inside the container. However, the same parent devices for the mount points can be seen using:

cat /proc/mounts

or

lsblk

Aerospike code uses /proc/mounts to get the parent device for the filesystem in use.

It should be noted that this warning should not happen in the case of using block devices directly (volumeMode: Block in the Aerospike Helm chart values.yaml file), because in that scenario the device path will be accessible directly in /dev/ inside the container.

In an environment where file based storage is being used, if the warnings are displayed with a high frequency it is most likely due to periodic info calls. Normally containers do not have access or direct visibility to the devices attached to the host. In a standard Docker setup, the --device option can be used to access the block device inside the container. Alternately the container can be run in --privileged mode which will allow the container to access all devices attached to the host.

In a Kubernetes managed environment, there is no option like --device however the container and pods can be run in privileged mode. So by deploying the aerospike statefulset with the container in privileged mode, this warning can be suppressed.

Solution

The current solution is to deploy the containers in privileged mode as shown below:

containers:

- name: aerospike

securityContext:

privileged: true

....

....

There is a caveat as this is not an ideal configuration from a security point of view. This is why this configuration is not the default.

Notes

  • More detail on deploying Docker within Kubernetes in privileged mode can be found at the link below:

  • The logging level of this message is removed as of version 4.9 (AER-6191). For earlier versions, to avoid excessive logging with the above WARNING message, the log level for the hardware context can be set to CRITICAL.

Keywords

KUBERNETES DOCKER HARDWARE FILE STORAGE-ENGINE NO SUCH FILE

Timestamp

January 2021