Hi, We’re running 3.5.15 on EC2 on i2.2xlarge. Recently we noticed that on some instances when the aerospike daemon is running, under light load, we start getting a lot of client timeouts and the cluster throughput decreases a lot. The ping time to the machine in question would increase substantial…

Btw, this happened again today on a different node in the cluster. UPDATE: 10/08 - it just happened on yet another instance. When this happens a very large number of the requests end up timing out so we end up removing that node from the cluster. Bringing it back up after a few hours or a day has t…

This happened again today on a different ec2 instance. @kporter , have you seen anything like this before? Any thoughts on what might help investigate this?

Couple of questions comes to mind. Are you using EBS for your storage? What’s the average object size? It would be interesting to enable micro-benchmark and storage-benchmark during these events. Please see: http://www.aerospike.com/docs/tools/asloglatency/#micro-benchmarks and http://www.aeros…

@lucien , This happened again, this time on a new node as soon as I added it to the cluster. We’re using SSD only, no EBS. Average object size should be pretty small - probably a few kb (our block size is 512k) - is there a stat for it somewhere? I saved the lsof output and it’s several MB so I’ve …

Looks like you are holding on to half open sockets. asd 28322 root 2027u sock 0,7 0t0 3479317 can't identify protocol asd 28322 root 2028u sock 0,7 0t0 3687513 can't identify protocol asd 28322 …

@lucien , The only place I see RecordSet used in the API is around query and we don’t use that - we only do (single and batch) put, get and execute. We only have 3 sets so we shouldn’t be querying any non-existing sets. Also, when I swap a node that starts having this problem with a new ec2 node the…

@lucien , One thing I was thinking about is that we’re not currently using VPC with Placement Group on ec2 as recommended by aerospike. Do you think this could be related to EC2 Classic network performance? We did have an AWS engineer look into one of the instances that showed this issue and they di…

@naoum , Are all the nodes on the same availability zone? At this point I can’t say for sure its a VPC issue, but being in the same availability zone would help stabilize the cluster.

@lucien , We are currently running aerospike on 2 AZ-s and we want to enable rack awareness but have not yet done that (we want to stabilize things first). The clients are on 4 AZ-s. Being on only 1 AZ does not sound like a great high availability strategy for when an AZ goes down.

There’s an interesting article on this topic of availability zone. Please see: [image] Are there recommendations for deploying on Amazon EC2? Configuration Are there recommendations for deploying on Amazon EC2?

Network throughput issues with asd running

lucien November 11, 2015, 4:44pm 11

There’s an interesting article on this topic of availability zone. Please see:

Topic		Replies	Views
Observing steady increase timeouts from aerospike server error	0	1752	March 10, 2016
Error in delay connects after system ready message Operations benchmark	11	3923	October 13, 2015
Node crash in AWS using the Aerospike AMI AWS ami , aws	8	5163	September 16, 2015
Hits AEROSPIKE_ERR_TIMEOUT more often than I should Operations	12	4070	May 14, 2015
One node showing inexplicably high read latency/CPU load Tuning aws , migration	10	5895	October 28, 2015

Network throughput issues with asd running

Related topics