Device mount warnings in docker

Hey,

I have aerospike running in docker - using compose and I am being spammed with the following every 5 seconds:

WARNING (hardware): (hardware.c:2262) failed to resolve mounted device /dev/mapper/ubuntu--vg-root: 2 (No such file or directory)

Does anyone know how to either

  • suppress this message (without disabling logging)
  • fix the underlying issue which may be causing this

My service definition is as follows:

aerospike:
image: aerospike:4.6.0.2
network_mode: "host"
command: ["--config-file" ,"/opt/aerospike/etc/aerospike.conf"]
volumes:
  - ./aerospike/data:/opt/aerospike/data
  - ./aerospike/etc:/opt/aerospike/etc

I’m not able to reproduce using your docker-compose.yml file. Were there any devicemapper driver configuration done on your host system running the containers. Please see: https://docs.docker.com/storage/storagedriver/device-mapper-driver/

Details on the repro attempt:

cat docker-compose.yml

version: '3.7'

services:


 aerospike:
     image: aerospike:4.6.0.2
     network_mode: "host"
     command: ["--config-file" ,"/opt/aerospike/etc/aerospike.conf"]
     volumes:
      - ./aerospike/data:/opt/aerospike/data
      - ./aerospike/etc:/opt/aerospike/etc
________________________________________________________________________________

| => docker-compose up -d
Starting dcompose_aerospike_1 ... done
________________________________________________________________________________

docker logs dcompose_aerospike_1

Sep 03 2019 19:44:17 GMT: INFO (as): (as.c:317) <><><><><><><><><><>  Aerospike Community Edition build 4.6.0.2  <><><><><><><><><><>
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) # Aerospike database configuration file.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) #
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) # This stanza must come first.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) service {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	user root
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	group root
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	service-threads 4
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	transaction-queues 4
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	transaction-threads-per-queue 4
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	proto-fd-max 15000
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) }
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) logging {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	# Send log messages to stdout
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	console {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		context any info 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) }
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) network {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	service {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		address eth0
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		port 3000
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# Uncomment the following to set the `access-address` parameter to the
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# IP address of the Docker host. This will the allow the server to correctly
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# publish the address which applications and other nodes in the cluster to
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# use when addressing this node.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# access-address <IPADDR>
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	heartbeat {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		address eth0
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# mesh is used for environments that do not support multicast
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		mode mesh
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		port 3002
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060)   #mesh-seed-address-port 10.0.2.2 3002
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# use asinfo -v 'tip:host=<ADDR>;port=3002' to inform cluster of
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# other mesh nodes
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		interval 150
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		timeout 10
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	fabric {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		address eth0
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		port 3001
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	info {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		port 3003
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) }
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) namespace test {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	replication-factor 2
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	memory-size 1G
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	default-ttl 5d # 5 days, use 0 to never expire/evict.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	#	storage-engine memory
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	# To use file storage backing, comment out the line above and use the
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	# following lines instead.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	storage-engine device {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		file /opt/aerospike/data/test.dat
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		filesize 4G
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		data-in-memory true # Store data in memory in addition to file.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) }
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4089) system file descriptor limit: 1048576, proto-fd-max: 15000
Sep 03 2019 19:44:17 GMT: INFO (hardware): (hardware.c:1991) detected 4 CPU(s), 4 core(s), 1 NUMA node(s)
Sep 03 2019 19:44:17 GMT: INFO (socket): (socket.c:2879) Node port 3001, node ID bb9010000005002
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4132) node-id bb9010000005002
Sep 03 2019 19:44:17 GMT: INFO (smd): (smd.c:2323) no file '/opt/aerospike/smd/evict.smd' - starting empty
Sep 03 2019 19:44:17 GMT: INFO (namespace): (namespace_ce.c:87) {test} beginning cold start
Sep 03 2019 19:44:17 GMT: INFO (smd): (smd.c:2323) no file '/opt/aerospike/smd/truncate.smd' - starting empty
Sep 03 2019 19:44:17 GMT: INFO (smd): (smd.c:2323) no file '/opt/aerospike/smd/sindex.smd' - starting empty
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:3339) opened file /opt/aerospike/data/test.dat: usable size 4294967296
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:1068) /opt/aerospike/data/test.dat has 4096 wblocks of size 1048576
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:2820) device /opt/aerospike/data/test.dat: reading device to load index
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:2829) device /opt/aerospike/data/test.dat: read complete: UNIQUE 0 (REPLACED 0) (OLDER 0) (EXPIRED 0) (EVICTED 0) records
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:1035) {test} loading free & defrag queues
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:972) /opt/aerospike/data/test.dat init defrag profile: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:1057) /opt/aerospike/data/test.dat init wblocks: pristine-id 8 pristine 4088 free-q 0, defrag-q 0
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:2189) {test} starting device maintenance threads
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:1495) {test} starting write threads
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:892) {test} starting defrag threads
Sep 03 2019 19:44:17 GMT: INFO (as): (as.c:376) initializing services...
Sep 03 2019 19:44:17 GMT: INFO (tsvc): (thr_tsvc.c:129) 4 transaction queues: starting 4 threads per queue
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:767) updated fabric published address list to {192.168.65.3:3001}
Sep 03 2019 19:44:17 GMT: INFO (partition): (partition_balance.c:201) {test} 4096 partitions: found 0 absent, 4096 stored
Sep 03 2019 19:44:17 GMT: INFO (hb): (hb.c:5525) updated heartbeat published address list to {192.168.65.3:3002}
Sep 03 2019 19:44:17 GMT: INFO (smd): (smd.c:2323) no file '/opt/aerospike/smd/UDF.smd' - starting empty
Sep 03 2019 19:44:17 GMT: INFO (batch): (batch.c:730) starting 4 batch-index-threads
Sep 03 2019 19:44:17 GMT: INFO (health): (health.c:319) starting health monitor thread
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:424) starting 8 fabric send threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:438) starting 16 fabric rw channel recv threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:438) starting 4 fabric ctrl channel recv threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:438) starting 4 fabric bulk channel recv threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:438) starting 4 fabric meta channel recv threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:444) starting fabric accept thread
Sep 03 2019 19:44:17 GMT: INFO (hb): (hb.c:7042) initializing mesh heartbeat socket: 192.168.65.3:3002
Sep 03 2019 19:44:17 GMT: INFO (fabric): (socket.c:847) Started fabric endpoint 192.168.65.3:3001
Sep 03 2019 19:44:17 GMT: INFO (hb): (hb.c:7071) mtu of the network is 1500
Sep 03 2019 19:44:17 GMT: INFO (hb): (socket.c:847) Started mesh heartbeat endpoint 192.168.65.3:3002
Sep 03 2019 19:44:17 GMT: INFO (nsup): (nsup.c:212) starting namespace supervisor threads
Sep 03 2019 19:44:17 GMT: INFO (service): (service.c:646) starting reaper thread
Sep 03 2019 19:44:17 GMT: INFO (service): (socket.c:847) Started client endpoint 192.168.65.3:3000
Sep 03 2019 19:44:17 GMT: INFO (service): (socket.c:847) Started client endpoint 127.0.0.1:3000
Sep 03 2019 19:44:17 GMT: INFO (service): (service.c:148) starting 4 service threads
Sep 03 2019 19:44:17 GMT: INFO (service): (service.c:157) starting accept thread
Sep 03 2019 19:44:17 GMT: INFO (info-port): (thr_info_port.c:298) starting info port thread
Sep 03 2019 19:44:17 GMT: INFO (info-port): (socket.c:847) Started info endpoint 0.0.0.0:3003
Sep 03 2019 19:44:17 GMT: INFO (as): (as.c:421) service ready: soon there will be cake!
Sep 03 2019 19:44:19 GMT: INFO (clustering): (clustering.c:6356) principal node - forming new cluster with succession list: bb9010000005002
Sep 03 2019 19:44:19 GMT: INFO (clustering): (clustering.c:5795) applied new cluster key 284554965a7
Sep 03 2019 19:44:19 GMT: INFO (exchange): (exchange.c:2315) data exchange started with cluster key 284554965a7
Sep 03 2019 19:44:19 GMT: INFO (clustering): (clustering.c:5797) applied new succession list bb9010000005002
Sep 03 2019 19:44:19 GMT: INFO (clustering): (clustering.c:5799) applied cluster size 1
Sep 03 2019 19:44:19 GMT: INFO (exchange): (exchange.c:3191) received commit command from principal node bb9010000005002
Sep 03 2019 19:44:19 GMT: INFO (exchange): (exchange.c:3154) data exchange completed with cluster key 284554965a7
Sep 03 2019 19:44:19 GMT: INFO (partition): (partition_balance.c:993) {test} replication factor is 1
Sep 03 2019 19:44:19 GMT: INFO (partition): (partition_balance.c:965) {test} rebalanced: expected-migrations (0,0,0) fresh-partitions 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:162) NODE-ID bb9010000005002 CLUSTER-SIZE 1
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:243)    cluster-clock: skew-ms 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:270)    system-memory: free-kbytes 1932344 free-pct 94 heap-kbytes (1091958,1092424,1112064) heap-efficiency-pct 98.2
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:282)    in-progress: tsvc-q 0 info-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:303)    fds: proto (0,0,0) heartbeat (0,0,0) fabric (0,0,0)
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:311)    heartbeat-received: self 0 foreign 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:341)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:399) {test} objects: all 0 master 0 prole 0 non-replica 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:457) {test} migrations: complete
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:475) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:527) {test} device-usage: used-bytes 0 avail-pct 99
Sep 03 2019 19:44:37 GMT: INFO (drv_ssd): (drv_ssd.c:1919) {test} /opt/aerospike/data/test.dat: used-bytes 0 free-wblocks 4088 write-q 0 write (0,0.0) defrag-q 0 defrag-read (0,0.0) defrag-write (0,0.0)
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:162) NODE-ID bb9010000005002 CLUSTER-SIZE 1
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:243)    cluster-clock: skew-ms 0
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:270)    system-memory: free-kbytes 1932268 free-pct 94 heap-kbytes (1091958,1092424,1112064) heap-efficiency-pct 98.2
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:282)    in-progress: tsvc-q 0 info-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:303)    fds: proto (0,0,0) heartbeat (0,0,0) fabric (0,0,0)
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:311)    heartbeat-received: self 0 foreign 0
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:341)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:399) {test} objects: all 0 master 0 prole 0 non-replica 0
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:457) {test} migrations: complete
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:475) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:527) {test} device-usage: used-bytes 0 avail-pct 99
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:162) NODE-ID bb9010000005002 CLUSTER-SIZE 1
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:243)    cluster-clock: skew-ms 0
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:270)    system-memory: free-kbytes 1932112 free-pct 94 heap-kbytes (1091958,1092424,1112064) heap-efficiency-pct 98.2
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:282)    in-progress: tsvc-q 0 info-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:303)    fds: proto (0,0,0) heartbeat (0,0,0) fabric (0,0,0)
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:311)    heartbeat-received: self 0 foreign 0
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:341)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:399) {test} objects: all 0 master 0 prole 0 non-replica 0
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:457) {test} migrations: complete
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:475) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:527) {test} device-usage: used-bytes 0 avail-pct 99

Thank you for the reply.

It would appear that it randomly stopped upon restarting the containers a bunch of times, potentially then related something to do with longer running containers. What is interesting is that it will continue to function just fine otherwise.

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.