Our application (rpc proxy into aerospike db) there’s one
com.aerospike.client.AerospikeClient long living connection to the 10 node Aerospike cluster.
Now (in dev) the application is almost always idle, but the
tendThread in the
AerospikeClient is connecting to the cluster every
tendInterval (1 sec).
After a while (few minutes) the application gets
too many open files error and stops responding.
I can see increasing number of
ESTABLISTED connections to the AS cluster (one node, “master”), that grows to about 650 and stops there (the number grows in increments of 10 = size of the cluster every second). When the number of
ESTABLISTED connections reaches ~ 650, there starts to appear
CLOSE_WAIT connections and number of them increments in same way.
By changing the connection parameters
maxSocketIdle I am able to change the rate of growth of new connections, but the result is always the same - after some time the application hangs with “too many open files” and thousands of
I also found that doing full sequential scan on a set (thus connecting each node) “heals” the application - all the
CLOSE_WAIT connections are gone and number of
ESTABLISTED is reduced to ~10 (and then the process starts over…).
Details: java client: com.aerospike:aerospike-client:jar:4.0.0 (maven dependency)
jvm: java version "1.8.0_51" Java™ SE Runtime Environment (build 1.8.0_51-b16) Java HotSpot™ 64-Bit Server VM (build 25.51-b03, mixed mode)
OS: debian wheezy
aerospike server: Aerospike Community Edition build 3.12.1