# Connections stuck in CLOSE\_WAIT **URL:** \ **Category:** General Discussion\ **Created:** [February 13, 2015, 5:11pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954 "2015-02-13T17:11:44Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![cody](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/cody/32/150_2.png) [@cody](https://discuss.aerospike.com/u/cody)\ **Post date:** [February 13, 2015, 5:11pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/1 "2015-02-13T17:11:44Z")
I’m seeing a situation where over a short period of time connections count ramps up dramatically, and the connections seem to be stuck in CLOSE\_WAIT Here’s a sample timeline from graphite of [connections, timestamp] [871.0, 1423801310], [871.0, 1423801312], [871.0, 1423801314], [4094.0, 1423801316], [4094.0, 1423801318], [4094.0, 1423801320], [4094.0, 1423801322], [4094.0, 1423801324], [4094.0, 1423801326], [4094.0, 1423801328], [4094.0, 1423801330], [4094.0, 1423801332], [4094.0, 1423801334], [4094.0, 1423801336], [4094.0, 1423801338], [4094.0, 1423801340], [4094.0, 1423801342], [4094.0, 1423801344], [18460.0, 1423801346] The vast majority of those 18,000 connections were stuck in CLOSE\_WAIT on the server side: asd 9962 root 206u IPv4 3552341 0t0 TCP ip-XXX.ec2.XXX:3000-\>ip-XX.ec2.XXX:57412 (CLOSE\_WAIT) This has intermittently happened several times, and I was only able to resolve it by hup’ing asd. For what it’s worth, this is with the PHP client library. I see the following presentation recommending setting proto-fd-idle-ms to 10 seconds for php, but even that wouldn’t have helped in this situation, because the connection limit was hit in the space of a couple of seconds. > **[Configuring Aerospike - Part 1](https://www.slideshare.net/AerospikeDB/configuring-aerospike-part-1)** > > Basic concepts and high level configuration. This is a basic overview of the Aerospike database and presents an introduction to configuring the database servic… ---
**Author:** ![rbotzer](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rbotzer/32/2114_2.png) [@rbotzer](https://discuss.aerospike.com/u/rbotzer)\ **Post date:** [February 13, 2015, 5:55pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/2 "2015-02-13T17:55:18Z")
Hey Cody, Thanks for reporting the problem. I’d like to dig into this, and first I want to try and reproduce it consistently. Can you add information about which release of the PHP client are you using, the OS and version on the client side, and the version of the server? Are your scripts using persistent connections? Is this a webserver context (for example Nginx + PHP-FPM)? If so could you give the webserver configuration? Are you using shared memory for cluster tending? Could you specify what the aerospike.shm.\* values are in your php.ini? Thanks! Ronen ---
**Author:** ![cody](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/cody/32/150_2.png) [@cody](https://discuss.aerospike.com/u/cody)\ **Post date:** [February 13, 2015, 6:58pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/3 "2015-02-13T18:58:11Z")
``` aerospike php client version 3.3.9 Amazon Linux AMI release 2014.09 3.14.27-25.47.amzn1.x86_64 aerospike-server-community-3.4.1-el6 It looks like Aerospike::__construct does a persistent connection by default, which is what we're using. Yes, Nginx + PHP-FPM nginx-1.6.2-1.22.amzn1.x86_64 Differences from nginx base config: worker_rlimit_nofile 200000; tcp_nopush on; keepalive_timeout 0; open_file_cache max=200000 inactive=20s; open_file_cache_valid 30s; open_file_cache_min_uses 2; open_file_cache_errors on; gzip on; fastcgi_buffers 8 16k; fastcgi_buffer_size 32k; cat /etc/php.d/aerospike.ini: extension=aerospike.so aerospike.udf.lua_system_path=/opt/aerospike/client-php/sys-lua aerospike.udf.lua_user_path=/opt/aerospike/client-php/usr-lua ``` ---
**Author:** ![rbotzer](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rbotzer/32/2114_2.png) [@rbotzer](https://discuss.aerospike.com/u/rbotzer)\ **Post date:** [February 13, 2015, 7:29pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/4 "2015-02-13T19:29:41Z")
Alrighty. I’ll look into it. Can you please try to explicitly set a `true` second parameter to the constructor. If that doesn’t change things, can you try to make use of the shared-memory cluster tending? You would add to your php.ini: ```auto aerospike.shm.use=true ``` Ronen ---
**Author:** ![rbotzer](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rbotzer/32/2114_2.png) [@rbotzer](https://discuss.aerospike.com/u/rbotzer)\ **Post date:** [February 16, 2015, 9:05pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/5 "2015-02-16T21:05:56Z")
One more thing, how many PHP processes do you have at the point where you see 18k CLOSE\_WAITs? Can you quote your `php-fpm.conf`? ---
**Author:** ![cody](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/cody/32/150_2.png) [@cody](https://discuss.aerospike.com/u/cody)\ **Post date:** [February 17, 2015, 3:46pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/6 "2015-02-17T15:46:25Z")
The issue seems to happen at low load times of the day (ie middle of the night), so approx 150 php processes across all fronted machines. There was a very noticeable load difference switching to explicit false for persistent connections, so it seems fairly likely they were on. We’ll try shared memory cluster tending. Relevant portions of php-fm config are: pm = dynamic pm.max\_children = 150 pm.max\_requests = 500 pm.max\_spare\_servers = 50 pm.min\_spare\_servers = 10 pm.start\_servers = 30 ---
**Author:** ![rbotzer](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rbotzer/32/2114_2.png) [@rbotzer](https://discuss.aerospike.com/u/rbotzer)\ **Post date:** [February 17, 2015, 4:01pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/7 "2015-02-17T16:01:40Z")
Thanks for the details. We’re looking into it. ---
**Author:** ![vlad](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/vlad/32/253_2.png) [@vlad](https://discuss.aerospike.com/u/vlad)\ **Post date:** [June 12, 2015, 5:53pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/8 "2015-06-12T17:53:43Z")
Have exactly the same issue with Java client too. The Aerospike server is practically idle (waiting for production deployment), Aerospike client webapp is started but not getting any traffic. Server version 3.5.12, java client 3.1.2 Please let me know if you need more info or want me to make any configuration changes. ---
**Author:** ![rbotzer](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rbotzer/32/2114_2.png) [@rbotzer](https://discuss.aerospike.com/u/rbotzer)\ **Post date:** [July 15, 2015, 11:50pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/9 "2015-07-15T23:50:58Z")
Two separate issues here. If the server side is idle it may be reaping the connections due to under-utilization you will see CLOSE\_WAITs, but I don’t think it’ll be in the thousands, and if the client isn’t doing much with the server then it’s not really a problem. It’s a relatively expensive operation (in terms of CPU and time) to initialize the client. It needs to learn the cluster topology after it connects to the seed node, then open TCP connections to each of the newly discovered nodes. In most clients (Java, Go, C#, etc) we do this once, then hold onto the client and send all requests through it. PHP, Python, and Ruby usually approach web applications in a different way. Since traditionally the problem was memory leaks in the interpreted code, the ‘solution’ was to severely limit the number of request each process should handle, fork new ones, and kill the ones that maxed out their requests. I would first suggest to raise the `max_requests` value as much as possible, while monitoring the processes for their memory consumption. The less the aerospike object gets recycled (and with it opening and closing connections), the less this will occur. We intend to investigate this further, however. I’d rather it worked better with a ‘standard’ FPM configuration. ---
**Author:** ![rbotzer](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rbotzer/32/2114_2.png) [@rbotzer](https://discuss.aerospike.com/u/rbotzer)\ **Post date:** [July 16, 2015, 12:28am UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/10 "2015-07-16T00:28:19Z")
@cody is that PHP non-ZTS? Check if path the extension\_dir is something like `/usr/local/lib/php/extensions/no-debug-non-zts-20131226`. ---
**Author:** ![cody](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/cody/32/150_2.png) [@cody](https://discuss.aerospike.com/u/cody)\ **Post date:** [July 17, 2015, 4:35pm UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/11 "2015-07-17T16:35:36Z")
@rbotzer Yeah, it’s /usr/lib/php/extensions/no-debug-non-zts-20131226/ ---
**Author:** ![rbotzer](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rbotzer/32/2114_2.png) [@rbotzer](https://discuss.aerospike.com/u/rbotzer)\ **Post date:** [December 25, 2015, 9:59am UTC](https://discuss.aerospike.com/t/connections-stuck-in-close-wait/954/12 "2015-12-25T09:59:34Z")
Please try the new client [release 3.4.6](https://github.com/aerospike/aerospike-client-php/releases/tag/3.4.6) and read the [Configuration in a Web Server Context](https://github.com/aerospike/aerospike-client-php/blob/master/doc/README.md#configuration-in-a-web-server-context) section of the overview.