Error Code 11: Failed to connect to host(s): 127.0.0.1:3000 Error Code 11: java.net.ConnectException: Cannot assign requested address


#1

During some peak times, my client failed to connect to aerospike with Error Code 11 as below

ERROR [resin-port-127.0.0.1:6800-7625] (AirPriceUapiOperationImpl.java:856) - com.aerospike.client.AerospikeException$Connection: Error Code 11: Failed to connect to host(s): 
127.0.0.1:3000 Error Code 11: java.net.ConnectException: Cannot assign requested address
	at com.aerospike.client.cluster.Cluster.seedNodes(Cluster.java:400)
	at com.aerospike.client.cluster.Cluster.tend(Cluster.java:269)
	at com.aerospike.client.cluster.Cluster.waitTillStabilized(Cluster.java:234)
	at com.aerospike.client.cluster.Cluster.initTendThread(Cluster.java:144)
	at com.aerospike.client.AerospikeClient.<init>(AerospikeClient.java:208)
	at com.aerospike.client.AerospikeClient.<init>(AerospikeClient.java:149)
	at com.mycompany.webservices.websvc.AirPriceUapiOperationImpl.getPricedItinerary(AirPriceUapiOperationImpl.java:206)

I have tried with “telnet localhost 3000”, it works

I have also tried to look up the transactions

    [root@localhost ~]# grep trans_in_progress /var/log/aerospike/aerospike.log | tail
    Dec 13 2015 05:52:17 GMT: INFO (info): (thr_info.c::4843)    trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (11464, 44757055, 44745591) : hb (0, 0, 0) : fab (16, 16, 0)

    [root@localhost ~]# grep ERROR /var/log/aerospike/aerospike.log | tail
    nothing

As I check out the local ports…

[root@localhost ~]# sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768	61000

Max connections of the server is set to 2048

[root@wefly ~]# sysctl net.core.somaxconn=2048
net.core.somaxconn = 2048

What else could I try to tune?