# No of connection ramps up dramatically **URL:** \ **Category:** Java Client\ **Created:** [August 13, 2018, 6:25pm UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444 "2018-08-13T18:25:21Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Sonaiya\_Karthick](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@Sonaiya\_Karthick](https://discuss.aerospike.com/u/Sonaiya_Karthick)\ **Post date:** [August 13, 2018, 6:25pm UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/1 "2018-08-13T18:25:21Z")
Hi, We are using aerospike in our organization extensively. We are facing some issues frequently nowadays. no of connection to aerospike is increasing suddenly from clients. It is reaching 15K connection and aerospike node stopped working. We had to restart the aerospike service. Most of the connections are coming from one of our netty servers. We upgraded our client version from 3.3.4 to 4.1.8. But we are facing the same issue. . ![49%20PM](https://us1.discourse-cdn.com/flex019/uploads/aerospike/original/2X/0/024ab085f77c504f81c1e26f8355fee1b93949c6.png) ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [August 13, 2018, 7:15pm UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/2 "2018-08-13T19:15:50Z")
To mitigate the symptom, you could increase [proto-fd-max](https://www.aerospike.com/docs/reference/configuration/index.html#proto-fd-max) to 100000. ---
**Author:** ![Brian](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/brian/32/2211_2.png) [@Brian](https://discuss.aerospike.com/u/Brian)\ **Post date:** [August 13, 2018, 8:09pm UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/3 "2018-08-13T20:09:36Z")
ClientPolicy.maxConnsPerNode (default 300) defines the upper bound of connections allowed to each to node. It appears that maxConnsPerNode has been set to a much higher value on your netty server/aerospike client. ---
**Author:** ![Sonaiya\_Karthick](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@Sonaiya\_Karthick](https://discuss.aerospike.com/u/Sonaiya_Karthick)\ **Post date:** [August 15, 2018, 3:14am UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/5 "2018-08-15T03:14:23Z")
Increasing proto-fd-max will not solve the problem. Here most of the connections are in the CLOSE\_WAIT state. It is keep on increasing. Anyway I increased proto-fd-max to 30K from 15K. ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [August 15, 2018, 3:35am UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/6 "2018-08-15T03:35:37Z")
I believe Brian likely identified the root cause. ---
**Author:** ![Sonaiya\_Karthick](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@Sonaiya\_Karthick](https://discuss.aerospike.com/u/Sonaiya_Karthick)\ **Post date:** [August 15, 2018, 7:17pm UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/7 "2018-08-15T19:17:45Z")
We have not set any value to ClientPolicy.maxConnsPerNode. So It should be 300 by default per node. 90% of connections are close\_wait connections and our cluster is going down because of connections. We had to restart the service frequently. _Server Version_ : 3.6.4 _Client Veersion_: 4.1.8 ---
**Author:** ![Brian](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/brian/32/2211_2.png) [@Brian](https://discuss.aerospike.com/u/Brian)\ **Post date:** [August 15, 2018, 7:49pm UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/8 "2018-08-15T19:49:51Z")
Many sockets in close/wait state usually means there are lots of timeouts occurring. A client timeout will force the socket to close. If the socket wasn’t closed on a client timeout, data from the previous transaction could be received in the next transaction when the socket is reused. Some ways to solve this are: 1. Increase transaction timeout values (Policy.socketTimeout and Policy.totalTimeout), so that timeouts occur less frequently. 2. Increase the amount of total OS sockets available on your machines. See: [[Solved - 100% Working Code]- How to Increase the maximum number of tcp/ip connections in linux - Linux Tutorial - Wikitechy](https://www.wikitechy.com/tutorials/linux/how-to-increase-the-maximum-number-of-tcp-ip-connections-in-linux) 3. Add extra hardware resources (machines, network bandwidth, etc…) to increase capacity, thus alleviating bottlenecks/timeouts. Another reason for large numbers of closed sockets is `ClientPolicy.maxSocketIdle` (default 55 seconds) is greater than the server’s `proto-fd-idle-ms` (default 60 seconds). `ClientPolicy.maxSocketIdle` should always be less than `proto-fd-idle-ms`. ---
**Author:** ![Sonaiya\_Karthick](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@Sonaiya\_Karthick](https://discuss.aerospike.com/u/Sonaiya_Karthick)\ **Post date:** [August 15, 2018, 8:46pm UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/9 "2018-08-15T20:46:06Z")
Thanks @Brian. Will make changes and update you. I checked the logs from clients. We are getting **java.io.EOFException** in the client side. ``` c.z.a.d.m.MomentsMarketLoaderDelegate - Exception while iterating from aerospike record set for moments marketing com.aerospike.client.AerospikeException: Error Code -1: java.io.EOFException at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:139) ~[prod-adtech-adserver.jar:na] at com.aerospike.client.command.MultiCommand.execute(MultiCommand.java:75) ~[prod-adtech-adserver.jar:na] at com.aerospike.client.query.QueryExecutor$QueryThread.run(QueryExecutor.java:146) ~[prod-adtech-adserver.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_151] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151] Caused by: java.io.EOFException: null at com.aerospike.client.command.MultiCommand.readBytes(MultiCommand.java:223) ~[prod-adtech-adserver.jar:na] at com.aerospike.client.command.MultiCommand.parseResult(MultiCommand.java:88) ~[prod-adtech-adserver.jar:na] at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:84) ~[prod-adtech-adserver.jar:na] ... 5 common frames omitted ``` Can you explain why it is happening? I might be the root cause of the above issue. ---
**Author:** ![Brian](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/brian/32/2211_2.png) [@Brian](https://discuss.aerospike.com/u/Brian)\ **Post date:** [August 16, 2018, 1:14am UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/10 "2018-08-16T01:14:33Z")
EOFException means the server closed the connection. By default, the server will close any open connection that has been idle for 60 seconds. By default, the server will also close connections where the socket is idle for 10 seconds on a running query. Your iteration of the record set may be backed up and no socket reads are occurring for 10 seconds. It’s important to iterate through the record set as fast as possible. It’s also critical to catch all application errors during the query and call RecordSet.close() at the end. RecordSet.close() is required on both success and error paths. ---
**Author:** ![Brian](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/brian/32/2211_2.png) [@Brian](https://discuss.aerospike.com/u/Brian)\ **Post date:** [August 16, 2018, 5:48pm UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/11 "2018-08-16T17:48:30Z")
Another question. What is the socketTimeout and totalTimeout values set on that query? ---
**Author:** ![Sonaiya\_Karthick](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@Sonaiya\_Karthick](https://discuss.aerospike.com/u/Sonaiya_Karthick)\ **Post date:** [August 17, 2018, 3:44am UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/12 "2018-08-17T03:44:57Z")
> [@Brian](#): > > Another @Brian We are setting 5ms for socket and total timeout ClientPolicy clientPolicy = new ClientPolicy(); clientPolicy.timeout = config.getInt(“Aerospike.connectionTimeout”); ``` Policy policy = new Policy(); policy.totalTimeout = config.getInt("Aerospike.totalTimeout",5); policy.socketTimeout = policy.totalTimeout; clientPolicy.readPolicyDefault = policy; QueryPolicy queryPolicy = new QueryPolicy(); queryPolicy.totalTimeout = config.getInt("Aerospike.Freqcap.totalTimeout", 5); queryPolicy.socketTimeout = queryPolicy.totalTimeout; queryPolicy.recordQueueSize = config.getInt("Aerospike.Freqcap.RecordQueueSize", 65535); clientPolicy.queryPolicyDefault = queryPolicy; return clientPolicy; ``` and we are overwriting queryPolicy in another aerospike query by default query policy like below ``` ` return new AerospikeSource(MomentsMarketCache.class, statement, config.getBoolean("Aerospike.MomentsMarket.isUpdatable"), momentsMarketLoaderDelegate, new QueryPolicy(), config.getInt("Aerospike.MomentsMarket.refreshTimeInterval"));` ``` Actually We are doing 3 transaction such as 1 read and 2 query operation. The socket and total timeout for 1 read and 1 query operation is 5 ms and another query operation has a default query policy. ---
**Author:** ![Brian](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/brian/32/2211_2.png) [@Brian](https://discuss.aerospike.com/u/Brian)\ **Post date:** [August 17, 2018, 5:34am UTC](https://discuss.aerospike.com/t/no-of-connection-ramps-up-dramatically/5444/13 "2018-08-17T05:34:49Z")
A 5ms timeout on a query is extremely aggressive. A single record get retrieves a single record from a single node. A query is issued to all nodes in the cluster and the client assembles the records into a single record set stream. There is at least an order of magnitude more work for queries than single record gets. The server will close the connection if a query timeout occurs, so that is likely why you are getting the EOFExceptions. I suggest raising the query timeout to a value much much higher than it’s currently set.