Our application (rpc proxy into aerospike db) there’s one com.aerospike.client.AerospikeClient long living connection to the 10 node Aerospike cluster.
Now (in dev) the application is almost always idle, but the tendThread in the AerospikeClient is connecting to the cluster every tendInterval (1 sec).
After a while (few minutes) the application gets too many open files error and stops responding.
I can see increasing number of ESTABLISTED connections to the AS cluster (one node, “master”), that grows to about 650 and stops there (the number grows in increments of 10 = size of the cluster every second). When the number of ESTABLISTED connections reaches ~ 650, there starts to appear CLOSE_WAIT connections and number of them increments in same way.
By changing the connection parameters tendInterval, maxConnsPerNode and maxSocketIdle I am able to change the rate of growth of new connections, but the result is always the same - after some time the application hangs with “too many open files” and thousands of CLOSE_WAIT connections.
I also found that doing full sequential scan on a set (thus connecting each node) “heals” the application - all the CLOSE_WAIT connections are gone and number of ESTABLISTED is reduced to ~10 (and then the process starts over…).
I’m experiencing the same issue with same Java client and Aerospace Community Edition on a CentOS 6.8 Server. Can someone please comment on this issue?
In my case, when the AS server is idle, I’ve found that the CLOSE_WAIT connections are building up at the rate of 1 per second.
When I downgraded client version to 3.3.4, problem solved.
It sounds like the cluster tend request (1 second interval) is constantly failing. The tend connection for each node is closed/reopened when an error/timeout occurs.
The AerospikeClient log will show tend errors. Can you subscribe to the AerospikeClient log and provide the log results?
public class MyConsole implements Log.Callback {
public MyConsole() {
Log.setLevel(Log.Level.DEBUG);
Log.setCallback(this);
}
@Override
public void log(Log.Level level, String message) {
// Write log messages to the appropriate place.
String date = SimpleDateFormat.format(new Date());
System.out.println(date.toString() + ' ' + level.toString() + ' ' + message);
}
}
I think I have found the problem. Try modifying Connection.java, isClosed(), and recompile:
public boolean isClosed() {
// Old
// return lastUsed == 0;
// New
return socket.isClosed();
}
If “socket.isClosed()” solves your problem, I will create a new release with a more optimized solution (socket.isClosed() acquires a lock which I would like to avoid).
!