Device mount warnings in docker

Ewan_Walker · September 3, 2019, 12:24pm

Hey,

I have aerospike running in docker - using compose and I am being spammed with the following every 5 seconds:

WARNING (hardware): (hardware.c:2262) failed to resolve mounted device /dev/mapper/ubuntu--vg-root: 2 (No such file or directory)

Does anyone know how to either

suppress this message (without disabling logging)
fix the underlying issue which may be causing this

My service definition is as follows:

aerospike:
image: aerospike:4.6.0.2
network_mode: "host"
command: ["--config-file" ,"/opt/aerospike/etc/aerospike.conf"]
volumes:
  - ./aerospike/data:/opt/aerospike/data
  - ./aerospike/etc:/opt/aerospike/etc

lucien · September 3, 2019, 7:51pm

I’m not able to reproduce using your docker-compose.yml file. Were there any devicemapper driver configuration done on your host system running the containers. Please see: Use the Device Mapper storage driver | Docker Docs

Details on the repro attempt:

cat docker-compose.yml

version: '3.7'

services:


 aerospike:
     image: aerospike:4.6.0.2
     network_mode: "host"
     command: ["--config-file" ,"/opt/aerospike/etc/aerospike.conf"]
     volumes:
      - ./aerospike/data:/opt/aerospike/data
      - ./aerospike/etc:/opt/aerospike/etc
________________________________________________________________________________

| => docker-compose up -d
Starting dcompose_aerospike_1 ... done
________________________________________________________________________________

docker logs dcompose_aerospike_1

Sep 03 2019 19:44:17 GMT: INFO (as): (as.c:317) <><><><><><><><><><>  Aerospike Community Edition build 4.6.0.2  <><><><><><><><><><>
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) # Aerospike database configuration file.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) #
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) # This stanza must come first.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) service {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	user root
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	group root
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	service-threads 4
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	transaction-queues 4
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	transaction-threads-per-queue 4
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	proto-fd-max 15000
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) }
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) logging {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	# Send log messages to stdout
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	console {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		context any info 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) }
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) network {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	service {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		address eth0
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		port 3000
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# Uncomment the following to set the `access-address` parameter to the
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# IP address of the Docker host. This will the allow the server to correctly
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# publish the address which applications and other nodes in the cluster to
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# use when addressing this node.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# access-address <IPADDR>
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	heartbeat {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		address eth0
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# mesh is used for environments that do not support multicast
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		mode mesh
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		port 3002
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060)   #mesh-seed-address-port 10.0.2.2 3002
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# use asinfo -v 'tip:host=<ADDR>;port=3002' to inform cluster of
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		# other mesh nodes
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		interval 150
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		timeout 10
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	fabric {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		address eth0
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		port 3001
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	info {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		port 3003
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) }
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) namespace test {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	replication-factor 2
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	memory-size 1G
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	default-ttl 5d # 5 days, use 0 to never expire/evict.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	#	storage-engine memory
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	# To use file storage backing, comment out the line above and use the
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	# following lines instead.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	storage-engine device {
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		file /opt/aerospike/data/test.dat
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		filesize 4G
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 		data-in-memory true # Store data in memory in addition to file.
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 	}
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) }
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4060) 
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4089) system file descriptor limit: 1048576, proto-fd-max: 15000
Sep 03 2019 19:44:17 GMT: INFO (hardware): (hardware.c:1991) detected 4 CPU(s), 4 core(s), 1 NUMA node(s)
Sep 03 2019 19:44:17 GMT: INFO (socket): (socket.c:2879) Node port 3001, node ID bb9010000005002
Sep 03 2019 19:44:17 GMT: INFO (config): (cfg.c:4132) node-id bb9010000005002
Sep 03 2019 19:44:17 GMT: INFO (smd): (smd.c:2323) no file '/opt/aerospike/smd/evict.smd' - starting empty
Sep 03 2019 19:44:17 GMT: INFO (namespace): (namespace_ce.c:87) {test} beginning cold start
Sep 03 2019 19:44:17 GMT: INFO (smd): (smd.c:2323) no file '/opt/aerospike/smd/truncate.smd' - starting empty
Sep 03 2019 19:44:17 GMT: INFO (smd): (smd.c:2323) no file '/opt/aerospike/smd/sindex.smd' - starting empty
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:3339) opened file /opt/aerospike/data/test.dat: usable size 4294967296
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:1068) /opt/aerospike/data/test.dat has 4096 wblocks of size 1048576
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:2820) device /opt/aerospike/data/test.dat: reading device to load index
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:2829) device /opt/aerospike/data/test.dat: read complete: UNIQUE 0 (REPLACED 0) (OLDER 0) (EXPIRED 0) (EVICTED 0) records
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:1035) {test} loading free & defrag queues
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:972) /opt/aerospike/data/test.dat init defrag profile: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:1057) /opt/aerospike/data/test.dat init wblocks: pristine-id 8 pristine 4088 free-q 0, defrag-q 0
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:2189) {test} starting device maintenance threads
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:1495) {test} starting write threads
Sep 03 2019 19:44:17 GMT: INFO (drv_ssd): (drv_ssd.c:892) {test} starting defrag threads
Sep 03 2019 19:44:17 GMT: INFO (as): (as.c:376) initializing services...
Sep 03 2019 19:44:17 GMT: INFO (tsvc): (thr_tsvc.c:129) 4 transaction queues: starting 4 threads per queue
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:767) updated fabric published address list to {192.168.65.3:3001}
Sep 03 2019 19:44:17 GMT: INFO (partition): (partition_balance.c:201) {test} 4096 partitions: found 0 absent, 4096 stored
Sep 03 2019 19:44:17 GMT: INFO (hb): (hb.c:5525) updated heartbeat published address list to {192.168.65.3:3002}
Sep 03 2019 19:44:17 GMT: INFO (smd): (smd.c:2323) no file '/opt/aerospike/smd/UDF.smd' - starting empty
Sep 03 2019 19:44:17 GMT: INFO (batch): (batch.c:730) starting 4 batch-index-threads
Sep 03 2019 19:44:17 GMT: INFO (health): (health.c:319) starting health monitor thread
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:424) starting 8 fabric send threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:438) starting 16 fabric rw channel recv threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:438) starting 4 fabric ctrl channel recv threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:438) starting 4 fabric bulk channel recv threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:438) starting 4 fabric meta channel recv threads
Sep 03 2019 19:44:17 GMT: INFO (fabric): (fabric.c:444) starting fabric accept thread
Sep 03 2019 19:44:17 GMT: INFO (hb): (hb.c:7042) initializing mesh heartbeat socket: 192.168.65.3:3002
Sep 03 2019 19:44:17 GMT: INFO (fabric): (socket.c:847) Started fabric endpoint 192.168.65.3:3001
Sep 03 2019 19:44:17 GMT: INFO (hb): (hb.c:7071) mtu of the network is 1500
Sep 03 2019 19:44:17 GMT: INFO (hb): (socket.c:847) Started mesh heartbeat endpoint 192.168.65.3:3002
Sep 03 2019 19:44:17 GMT: INFO (nsup): (nsup.c:212) starting namespace supervisor threads
Sep 03 2019 19:44:17 GMT: INFO (service): (service.c:646) starting reaper thread
Sep 03 2019 19:44:17 GMT: INFO (service): (socket.c:847) Started client endpoint 192.168.65.3:3000
Sep 03 2019 19:44:17 GMT: INFO (service): (socket.c:847) Started client endpoint 127.0.0.1:3000
Sep 03 2019 19:44:17 GMT: INFO (service): (service.c:148) starting 4 service threads
Sep 03 2019 19:44:17 GMT: INFO (service): (service.c:157) starting accept thread
Sep 03 2019 19:44:17 GMT: INFO (info-port): (thr_info_port.c:298) starting info port thread
Sep 03 2019 19:44:17 GMT: INFO (info-port): (socket.c:847) Started info endpoint 0.0.0.0:3003
Sep 03 2019 19:44:17 GMT: INFO (as): (as.c:421) service ready: soon there will be cake!
Sep 03 2019 19:44:19 GMT: INFO (clustering): (clustering.c:6356) principal node - forming new cluster with succession list: bb9010000005002
Sep 03 2019 19:44:19 GMT: INFO (clustering): (clustering.c:5795) applied new cluster key 284554965a7
Sep 03 2019 19:44:19 GMT: INFO (exchange): (exchange.c:2315) data exchange started with cluster key 284554965a7
Sep 03 2019 19:44:19 GMT: INFO (clustering): (clustering.c:5797) applied new succession list bb9010000005002
Sep 03 2019 19:44:19 GMT: INFO (clustering): (clustering.c:5799) applied cluster size 1
Sep 03 2019 19:44:19 GMT: INFO (exchange): (exchange.c:3191) received commit command from principal node bb9010000005002
Sep 03 2019 19:44:19 GMT: INFO (exchange): (exchange.c:3154) data exchange completed with cluster key 284554965a7
Sep 03 2019 19:44:19 GMT: INFO (partition): (partition_balance.c:993) {test} replication factor is 1
Sep 03 2019 19:44:19 GMT: INFO (partition): (partition_balance.c:965) {test} rebalanced: expected-migrations (0,0,0) fresh-partitions 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:162) NODE-ID bb9010000005002 CLUSTER-SIZE 1
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:243)    cluster-clock: skew-ms 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:270)    system-memory: free-kbytes 1932344 free-pct 94 heap-kbytes (1091958,1092424,1112064) heap-efficiency-pct 98.2
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:282)    in-progress: tsvc-q 0 info-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:303)    fds: proto (0,0,0) heartbeat (0,0,0) fabric (0,0,0)
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:311)    heartbeat-received: self 0 foreign 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:341)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:399) {test} objects: all 0 master 0 prole 0 non-replica 0
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:457) {test} migrations: complete
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:475) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Sep 03 2019 19:44:27 GMT: INFO (info): (ticker.c:527) {test} device-usage: used-bytes 0 avail-pct 99
Sep 03 2019 19:44:37 GMT: INFO (drv_ssd): (drv_ssd.c:1919) {test} /opt/aerospike/data/test.dat: used-bytes 0 free-wblocks 4088 write-q 0 write (0,0.0) defrag-q 0 defrag-read (0,0.0) defrag-write (0,0.0)
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:162) NODE-ID bb9010000005002 CLUSTER-SIZE 1
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:243)    cluster-clock: skew-ms 0
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:270)    system-memory: free-kbytes 1932268 free-pct 94 heap-kbytes (1091958,1092424,1112064) heap-efficiency-pct 98.2
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:282)    in-progress: tsvc-q 0 info-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:303)    fds: proto (0,0,0) heartbeat (0,0,0) fabric (0,0,0)
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:311)    heartbeat-received: self 0 foreign 0
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:341)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:399) {test} objects: all 0 master 0 prole 0 non-replica 0
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:457) {test} migrations: complete
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:475) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Sep 03 2019 19:44:37 GMT: INFO (info): (ticker.c:527) {test} device-usage: used-bytes 0 avail-pct 99
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:162) NODE-ID bb9010000005002 CLUSTER-SIZE 1
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:243)    cluster-clock: skew-ms 0
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:270)    system-memory: free-kbytes 1932112 free-pct 94 heap-kbytes (1091958,1092424,1112064) heap-efficiency-pct 98.2
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:282)    in-progress: tsvc-q 0 info-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:303)    fds: proto (0,0,0) heartbeat (0,0,0) fabric (0,0,0)
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:311)    heartbeat-received: self 0 foreign 0
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:341)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:399) {test} objects: all 0 master 0 prole 0 non-replica 0
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:457) {test} migrations: complete
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:475) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Sep 03 2019 19:44:47 GMT: INFO (info): (ticker.c:527) {test} device-usage: used-bytes 0 avail-pct 99

Ewan_Walker · September 4, 2019, 1:46pm

Thank you for the reply.

It would appear that it randomly stopped upon restarting the containers a bunch of times, potentially then related something to do with longer running containers. What is interesting is that it will continue to function just fine otherwise.

Topic		Replies	Views
Starting Aerospike EE on docker throws Exited (1) error Containers (Docker, etc.)	2	2466	September 3, 2019
Aerospike docker connection refused using docker servicename	1	1096	November 30, 2019
How to troubleshoot warnings in Aerospike logs after node restart Configuration	1	1032	December 1, 2020
Aerospike server in docker container abnormal exit (Panic) Containers (Docker, etc.)	4	1866	August 8, 2017
Error when running the latest Node.js release Node.js Client container , docker	3	2051	May 11, 2015

Device mount warnings in docker

Related Topics