# Java client - tendThread leaves connections in CLOSE\_WAIT state **URL:** \ **Category:** Java Client\ **Created:** [May 30, 2017, 6:30am UTC](https://discuss.aerospike.com/t/java-client-tendthread-leaves-connections-in-close-wait-state/4207 "2017-05-30T06:30:12Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jindrich.vimr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/jindrich.vimr/32/789_2.png) [@jindrich.vimr](https://discuss.aerospike.com/u/jindrich.vimr)\ **Post date:** [May 30, 2017, 6:30am UTC](https://discuss.aerospike.com/t/java-client-tendthread-leaves-connections-in-close-wait-state/4207/1 "2017-05-30T06:30:12Z")
Our application (rpc proxy into aerospike db) there’s one `com.aerospike.client.AerospikeClient` long living connection to the 10 node Aerospike cluster. Now (in dev) the application is almost always idle, but the `tendThread` in the `AerospikeClient` is connecting to the cluster every `tendInterval` (1 sec). After a while (few minutes) the application gets `too many open files` error and stops responding. I can see increasing number of `ESTABLISTED` connections to the AS cluster (one node, “master”), that grows to about 650 and stops there (the number grows in increments of 10 = size of the cluster every second). When the number of `ESTABLISTED` connections reaches ~ 650, there starts to appear `CLOSE_WAIT` connections and number of them increments in same way. By changing the connection parameters `tendInterval`, `maxConnsPerNode` and `maxSocketIdle` I am able to change the rate of growth of new connections, but the result is always the same - after some time the application hangs with “too many open files” and thousands of `CLOSE_WAIT` connections. I also found that doing full sequential scan on a set (thus connecting each node) “heals” the application - all the `CLOSE_WAIT` connections are gone and number of `ESTABLISTED` is reduced to ~10 (and then the process starts over…). Details: java client: com.aerospike:aerospike-client:jar:4.0.0 (maven dependency) jvm: java version “1.8.0\_51” Java™ SE Runtime Environment (build 1.8.0\_51-b16) Java HotSpot™ 64-Bit Server VM (build 25.51-b03, mixed mode) OS: debian wheezy aerospike server: Aerospike Community Edition build 3.12.1 ---
**Author:** ![sonerx](https://avatars.discourse-cdn.com/v4/letter/s/ac91a4/32.png) [@sonerx](https://discuss.aerospike.com/u/sonerx)\ **Post date:** [May 30, 2017, 6:49pm UTC](https://discuss.aerospike.com/t/java-client-tendthread-leaves-connections-in-close-wait-state/4207/2 "2017-05-30T18:49:46Z")
I’m experiencing the same issue with same Java client and Aerospace Community Edition on a CentOS 6.8 Server. Can someone please comment on this issue? In my case, when the AS server is idle, I’ve found that the CLOSE\_WAIT connections are building up at the rate of 1 per second. When I downgraded client version to 3.3.4, problem solved. ---
**Author:** ![Brian](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/brian/32/2211_2.png) [@Brian](https://discuss.aerospike.com/u/Brian)\ **Post date:** [May 30, 2017, 9:24pm UTC](https://discuss.aerospike.com/t/java-client-tendthread-leaves-connections-in-close-wait-state/4207/3 "2017-05-30T21:24:50Z")
It sounds like the cluster tend request (1 second interval) is constantly failing. The tend connection for each node is closed/reopened when an error/timeout occurs. The AerospikeClient log will show tend errors. Can you subscribe to the AerospikeClient log and provide the log results? ```java public class MyConsole implements Log.Callback { public MyConsole() { Log.setLevel(Log.Level.DEBUG); Log.setCallback(this); } @Override public void log(Log.Level level, String message) { // Write log messages to the appropriate place. String date = SimpleDateFormat.format(new Date()); System.out.println(date.toString() + ' ' + level.toString() + ' ' + message); } } ``` ---
**Author:** ![Brian](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/brian/32/2211_2.png) [@Brian](https://discuss.aerospike.com/u/Brian)\ **Post date:** [May 30, 2017, 9:58pm UTC](https://discuss.aerospike.com/t/java-client-tendthread-leaves-connections-in-close-wait-state/4207/4 "2017-05-30T21:58:20Z")
I think I have found the problem. Try modifying Connection.java, isClosed(), and recompile: ```java public boolean isClosed() { // Old // return lastUsed == 0; // New return socket.isClosed(); } ``` If “socket.isClosed()” solves your problem, I will create a new release with a more optimized solution (socket.isClosed() acquires a lock which I would like to avoid). ---
**Author:** ![Brian](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/brian/32/2211_2.png) [@Brian](https://discuss.aerospike.com/u/Brian)\ **Post date:** [May 31, 2017, 2:06am UTC](https://discuss.aerospike.com/t/java-client-tendthread-leaves-connections-in-close-wait-state/4207/5 "2017-05-31T02:06:28Z")
Java client 4.0.1 has been released: > **[Aerospike Downloads](https://aerospike.com/download/#aerospike-clients-java-client-library)** > > Get the latest, all supported, and archived Aerospike software downloads here. The cluster tend connection should now stay open through successive intervals. ---
**Author:** ![jindrich.vimr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/jindrich.vimr/32/789_2.png) [@jindrich.vimr](https://discuss.aerospike.com/u/jindrich.vimr)\ **Post date:** [May 31, 2017, 6:21am UTC](https://discuss.aerospike.com/t/java-client-tendthread-leaves-connections-in-close-wait-state/4207/6 "2017-05-31T06:21:31Z")
I’ve tried the Java client 4.0.1 and yes, it solves the problem 🙂 ! I’ve also subscribed to the log messages (in 4.0.0 version), but there was no logged message. Thanks for such a quick fix! ---
**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\ **Post date:** [June 1, 2017, 1:50pm UTC](https://discuss.aerospike.com/t/java-client-tendthread-leaves-connections-in-close-wait-state/4207/7 "2017-06-01T13:50:40Z")
Also see: [Aerospike: Invalid node exception when writing to single node Aerospike. - Stack Overflow](https://stackoverflow.com/questions/44259941/aerospike-invalid-node-exception-when-writing-to-single-node-aerospike)