Swarm and Stop writes

Hi,

I install a cluster with Docker swarm. First i load with 2 db and next i test with a sclate to 8 db. With 2 db, load generate errors quickly and stop because all db are full. With 8 db, load also generate errors but only when data flow is sent to db full (1 or 2). My data is a file of 50MB.

With Asadm i can get stop writes error at true with Available 1 or 0 % with 15-20MB used on disk. At the load beginning, Available was at 99%.

I used params aerospike indicated in this blog

Docker machine create per default a node with 1GB of ram, i tried to increase to 1,5GB but there is no effects. I have 6GB of free RAM (10 are used) and 400GB free on my SSD.

How i can allow my system to load correctly data ?

Thanks

What version is the server?

Could you share your /etc/aerospike.conf?

i’m using the last version of server present on docker hub : 4.2.0.5

You will find bellow the content of file .conf upload by yml on all containers :
# Aerospike database configuration file.

# This stanza must come first.
service {
	user root
	group root
	paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
	pidfile /var/run/aerospike/asd.pid
	service-threads 4
	transaction-queues 4
	transaction-threads-per-queue 4
	proto-fd-max 15000
}

logging {

	# Log file must be an absolute path.
	file /var/log/aerospike/aerospike.log {
		context any info
	}

	# Send log messages to stdout
	console {
		context any info 
	}
}

network {
	service {
		address eth0
		port 3000

		# Uncomment the following to set the `access-address` parameter to the
		# IP address of the Docker host. This will the allow the server to correctly
		# publish the address which applications and other nodes in the cluster to
		# use when addressing this node.
		# access-address <IPADDR>
	}

	heartbeat {
		address eth0
		# mesh is used for environments that do not support multicast
		mode mesh
		port 3002

		# use asinfo -v 'tip:host=<ADDR>;port=3002' to inform cluster of
		# other mesh nodes

		interval 150
		timeout 10
	}

	fabric {
		address eth0
		port 3001
	}

	info {
		port 3003
	}
}

namespace test {
	replication-factor 2
	memory-size 1G
	default-ttl 5d # 5 days, use 0 to never expire/evict.

	#	storage-engine memory

	# To use file storage backing, comment out the line above and use the
	# following lines instead.
	storage-engine device {
		file /opt/aerospike/data/test.dat
		filesize 4G
		data-in-memory true # Store data in memory in addition to file.
	}
}
namespace nfe204 {
	replication-factor 2
	memory-size 1G
	default-ttl 5d # 5 days, use 0 to never expire/evict.

	#	storage-engine memory

	# To use file storage backing, comment out the line above and use the
	# following lines instead.
	storage-engine device {
		file /opt/aerospike/data/nfe204.dat
		filesize 4G
		data-in-memory true # Store data in memory in addition to file.
	}
}

But if i access to my container, file /etc/aerospike.conf is not present.

I can find file in /etc/aerospike/aerospike.conf .

But the content seem like to be the default conf file (new namespace is not present) but with asadm he is listed.

If you followed the blog post, your conf file exists as a Docker secret at /run/secrets/aerospike.conf and upon Aerospike container start, we utilize the --config-file param to utilize this secret.

Stop-writes can occur when device_available_pct drops below min-avail-pct, or when memory is over 90% used.

Can you please post the output from asadm -e info?

Ok, conf present in /run/secrets is the same that indicated.

Here is asadm -e info after increase memory of each nodes to 2.5GB :

I succeed to load only 28% of my file.

At the surface, it looks like defragmentation is unable to keep up.

If you’ve also used this loader to load in the data, then you should not have defragmentation at all, as all data are written sequentially (no updates/deletes).

We can find out more with some log lines. About 50 or so of the last lines from the log is enough. Either /var/log/aerospike/aerospike.log inside of the container, or docker logs --tail 50 {aerospike_container}

Lastly, are all 4 docker nodes/virtual machines on a single machine? Like your laptop?

Hi,

This is a part of service db logs :

aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: WARNING (drv_ssd): (drv_ssd.c:681) {nfe204}: defrag: drive /opt/aerospike/data/nfe204.dat totally full, re-queuing wblock 4091
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:171) NODE-ID bb91b01000a4202 CLUSTER-SIZE 2
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:277)    system-memory: free-kbytes 5919472 free-pct 91 heap-kbytes (2201122,2202556,2277376) heap-efficiency-pct 96.7
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:291)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:247)    cluster-clock: skew-ms 134
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:277)    system-memory: free-kbytes 5579596 free-pct 86 heap-kbytes (2249654,2253068,2533376) heap-efficiency-pct 88.8
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:313)    fds: proto (0,5,5) heartbeat (1,4,3) fabric (24,60,36)
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:322)    heartbeat-received: self 2 foreign 498
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:291)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:313)    fds: proto (0,60,60) heartbeat (1,3,2) fabric (24,24,0)
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:353)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:408) {test} objects: all 0 master 0 prole 0 non-replica 0
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:322)    heartbeat-received: self 2 foreign 5745
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:353)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:469) {test} migrations: complete
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:488) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:408) {test} objects: all 0 master 0 prole 0 non-replica 0
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:469) {test} migrations: complete
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:518) {test} device-usage: used-bytes 0 avail-pct 99
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:408) {nfe204} objects: all 0 master 0 prole 0 non-replica 0
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:488) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:518) {test} device-usage: used-bytes 0 avail-pct 99
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:469) {nfe204} migrations: complete
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:488) {nfe204} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:408) {nfe204} objects: all 259143 master 165547 prole 93596 non-replica 0
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:469) {nfe204} migrations: complete
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: INFO (info): (ticker.c:518) {nfe204} device-usage: used-bytes 0 avail-pct 99
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: WARNING (socket): (socket.c:755) (repeated:1) Error while connecting: 113 (No route to host)
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:488) {nfe204} memory-usage: total-bytes 41598109 index-bytes 16585152 sindex-bytes 0 data-bytes 25012957 used-pct 3.87
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:518) {nfe204} device-usage: used-bytes 41054560 avail-pct 0
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: WARNING (socket): (socket.c:746) (repeated:65) Timeout while connecting
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: WARNING (hb): (hb.c:4845) (repeated:66) could not create heartbeat connection to node {10.0.1.19:3002}
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (ticker.c:587) {nfe204} client: tsvc (0,13) proxy (334114,0,4) read (0,0,0,0) write (119520,121663,0) delete (0,0,0,0) udf (0,0,0) lang (0,0,0,0)
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (hist.c:240) histogram dump: {nfe204}-write (241183 total) msec
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: WARNING (socket): (socket.c:814) (repeated:66) Error while connecting socket to 10.0.1.19:3002
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:22 GMT: INFO (as): (signal.c:194) SIGTERM received, starting normal shutdown
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (hist.c:257)  (00: 0000141864) (01: 0000026791) (02: 0000023802) (03: 0000009846)
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (hist.c:257)  (04: 0000007177) (05: 0000010088) (06: 0000008994) (07: 0000007115)
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:22 GMT: INFO (storage): (storage.c:702) initiating storage shutdown ...
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:22 GMT: INFO (storage): (storage.c:703) flushing data to storage ...
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:38 GMT: INFO (info): (hist.c:266)  (08: 0000003821) (09: 0000001480) (10: 0000000205)
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:39 GMT: WARNING (drv_ssd): (drv_ssd.c:681) {nfe204}: defrag: drive /opt/aerospike/data/nfe204.dat totally full, re-queuing wblock 4091
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:22 GMT: INFO (storage): (storage.c:722) completed flushing to storage
aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:22 GMT: INFO (as): (as.c:445) finished clean shutdown - exiting
aerospike_aerospikedb.1.rsbys4zssn6j@nodeleader-1    | Aug 01 2018 21:33:40 GMT: WARNING (drv_ssd): (drv_ssd.c:681) {nfe204}: defrag: drive /opt/aerospike/data/nfe204.dat totally full, re-queuing wblock 4091
error from daemon in stream: Error grabbing logs: rpc error: code = Unknown desc = warning: incomplete log stream. some logs could not be retrieved for the following reasons: node g928tx0jgmotip7z5xygvr9mo is not available

Errors seems to come from a prob of disk and not memory.

I use default config and i don’t understand how my storage file can be full with a so little data flow.

All my nodes/VM are on the same desktop with a SSD NVMe of 500GB.

From the write histogram you provided, writes seem to be taking a long time to complete. Not sure if it is your disk or not, could you configure enable-benchmarks-storage to true and rerun this test. More then 10% took over 32 ms and some even took more than 1 second.

Also defrag is unable to keep up which would also indicate that the disk is not able to handle the load.

Why are there so many proxies (proxy (334114,0,4))? Proxies should be relatively rare, with spike around cluster disruptions. If you are proxying all the time then a client may not be able to see all nodes.

Why is this node having trouble connecting to the other nodes?

aerospike_aerospikedb.1.sgzk1ai0f95b@nodeleader-1    | Aug 01 2018 21:17:18 GMT: WARNING (socket): (socket.c:746) (repeated:65) Timeout while connecting

Are there supposed to be 4 nodes in this cluster? Only two of them are forming a cluster.

Docker and friends add bring a lot of complexity into the system, I would suggest trying to get things running without the extra complexity first. Also have you verified that your disks can handle to load you using ACT?

A few things going on here.

  1. There’s 2 container’s worth of logs, 15 minutes apart, interwoven. It seems like you’ve deleted the entire cluster and recreated it, which is why they’re both aerospike_aerospikedb.1, but the next string is different, rsby... vs sgzk....
  2. Proxies/Load Balancers should not be used with Aerospike. Unless you’re using the very latest Java and C clients and even then, it’s only useful for discovery, not actual traffic routing.

So if you’re deploying Aerospike onto containers (Docker, Kubernetes, Mesos/Marathon, etc…), you’d either need to use host-based networking, or place your clients inside of the same container environment.

My purpose is to create a cluster for a school project. I use this loader

I think i have error to connect because each time i reboot my destop, services are randomly redispatch between nodes up.

On my desktop, i have installed Aero DB and load without error all my data flow.

I use Haproxy to interact between my os and Docker. My docker config is precised in this topic

On my cluster with more memory, i up to 7 nodes and load.

The load has finish without stop writes but 2/3 of my data are in errors.

ERROR AsWriterTask     :161  - File: temp_state.csv Line: 185807Aerospike Write Error: 18
... 
ERROR AsWriterTask     :161  - File: temp_state.csv Line: 188082Aerospike Write Error: 8
...
ERROR AsWriterTask     :161  - File: temp_state.csv Line: 188082Aerospike Write Error: 9

INFO AerospikeLoad :430 - Final Statistics of importer: (Records Read = 645675, Successful Writes = 481719, Successful Primary Writes = 259136, Successful Mapping Writes = 222583, Errors = 809631(809631-Write,0-Read,0-Processing), Skipped = 0(0-NullKey,0-NoBins)

I think i have error to connect because each time i reboot my destop, services are randomly redispatch between nodes

If everything’s on a single machine, then all containers will be placed in a stopped state, able to be and are resumed after reboot. If your Docker Swarm is across multiple machines, then the containers that were brought down with the host will be removed and new replacements created on the surviving host(s).

I use Haproxy to interact between my os and Docker

Aerospike does not work correctly with proxies/load balancers. Each Aerospike Client keeps track of which server node to place each record. Clients connect directly to the server’s published address. By forcing a proxied route, client connections cannot reach server nodes directly, and must timeout and attempt the proxy. The proxy will then forward the connection to a random server node, which the server most likely then has to forward to the correct server node.

You’re probably seeing loader write files due to the above process taking too long and timing out.

So then how can you communicate with Aerospike inside of Docker?

By placing your clients inside of the same Docker environment. Build a simple container with your compiled client code and run it:

FROM: debian:stretch
COPY . /root
ENTRYPOINT /root/<my-client-binary>

Also you’re changing your cluster each time, going from 2 nodes, to 4 nodes, to 7 nodes. Please stop doing that. If your dataset is only 50MB, then stick with 2 nodes and in-memory only. Once you’ve figured out how networking should be done, then add persistent storage.