I have a very basic cluster:
something similar to that of
version: '3' services: aerospike: image: aerospike:126.96.36.199 network_mode: "host" command: ["--config-file" ,"/opt/aerospike/etc/aerospike.conf"] volumes: - ./data:/opt/aerospike/data - ./etc:/opt/aerospike/etc ulimits: nofile: soft: "65536" hard: "65536" logging: driver: json-file options: max-size: 10m max-file: "10"
Which is running on two separate servers. If run
docker-compose down --rmi all and start the clusters they seem to work and join however are somehow forever doing migrations (with no data?).
My problems are:
- Somehow I can start a cluster with size 0 ?
- Good chance of seeing the following being spammed in the logs:
WARNING (hardware): (hardware.c:2262) failed to resolve mounted device /dev/md1: 2 (No such file or directory)
- Restarting a single instance essentially breaks the cluster and no longer rejoins citing:
skipping forming cluster - cannot form new cluster from pending join requests (empty)
join request timed out for principal bb96c64c0902500
ctrl ack (14): unexpected source bb96c64c0902500
The only thing which resolves this is completely deleting the image (so far) and trying again - which is not production friendly.
The other issue I see is that migrations take quite literally forever - even though there is no data in the cluster at all.
The servers are running SSDs, my configuration is pretty much default (like this https://github.com/aerospike/aerospike-server/blob/master/as/etc/aerospike.conf)
What am I missing?