Dropping incoming conneсtions


#1

Hi,

We’ve faced an issue today:

Dec 27 2016 13:59:04 GMT: WARNING (demarshal): (thr_demarshal.c:446) dropping incoming client connection: hit limit 150001 connections

while netstat showed not more than 1k connections to aerospike

we are using aerospike-server-community 3.10.1-1

How is it possible?


#2

Which client/version are you using (Java, Python, etc)?


#3

C Client Library 4.1.2


#4

We again noticed same issue:

Dec 28 2016 08:28:49 GMT: WARNING (demarshal): (thr_demarshal.c:446) dropping incoming client connection: hit limit 190001 connections

while asadm shows 1500 connections (same as netstat)

the host is extremely slow during that time and whole cluster experience issues with latency till aerospike restarted on this node.

we have 4 nodes cluster with replication-factor 2


#5

Was anything recently updated (client or server)?

Is this a new usecase? A common mistake is closing and reopening the aerospike client for each request. The aerospike clients are thread safe and should only need one for an application.


#6

We think we found a bug at our client side. Thanks for help