# Exception on node restart

**URL:** <https://discuss.aerospike.com/t/exception-on-node-restart/4724>\
**Category:** General Discussion\
**Created:** [October 30, 2017, 2:08pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724 "2017-10-30T14:08:41Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![dhanasekaran1980](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@dhanasekaran1980](https://discuss.aerospike.com/u/dhanasekaran1980)\
**Post date:** [October 30, 2017, 2:08pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/1 "2017-10-30T14:08:41Z")

</div>

Hi, I am using Aerospike 3.15.0.1 with Java 4.0.8 client. Whenever I restart a node from 3 node cluster, I am getting below exception and the request is not routed to other node. I have configured all 3 nodes IPs at client side.

```
com.aerospike.client.AerospikeException: java.io.EOFException
Caused by: java.io.EOFException
com.aerospike.client.AerospikeException: Error Code 11: Server not available

```

My namespace configuration is

```
namespace dm {
        replication-factor 2
        memory-size 5G
        default-ttl 1d # 1 days, use 0 to never expire/evict.
        storage-engine memory
}

```

Please help.

---

<div class="post-metadata">

**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\
**Post date:** [October 30, 2017, 3:39pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/2 "2017-10-30T15:39:02Z")

</div>

what is your client side code for opening connections?

---

<div class="post-metadata">

**Author:** ![dhanasekaran1980](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@dhanasekaran1980](https://discuss.aerospike.com/u/dhanasekaran1980)\
**Post date:** [October 30, 2017, 4:02pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/3 "2017-10-30T16:02:20Z")

</div>

Gupta, Appreciate your quick reply. Below is my code snippet

```
clientPolicy.maxConnsPerNode = 100;
clientPolicy.maxSocketIdle = 60;
clientPolicy.timeout = 5000;

client = new AerospikeClient(clientPolicy, hosts);
writePolicy = new WritePolicy();
try{
    writePolicy.expiration = 300;
    writePolicy.recordExistsAction = RecordExistsAction.CREATE_ONLY;
    bin1 = new Bin(null, k);
    client.put(writePolicy, key, bin1);
  }catch(AerospikeException e){
      if(e.getResultCode() == ResultCode.KEY_EXISTS_ERROR){
          writePolicy.expiration = -2;
          writePolicy.recordExistsAction = RecordExistsAction.UPDATE_ONLY;  
          bin1 = new Bin(null, 1);
          Record record = client.operate(writePolicy, key, Operation.add(bin1), Operation.get());
      }
}

```

---

<div class="post-metadata">

**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\
**Post date:** [October 30, 2017, 4:08pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/4 "2017-10-30T16:08:00Z")

</div>

I would recommend maxSocketIdle to 55 seconds instead of 60. Server side proto-fd-idle-max is 60000 ms or 60 seconds. what is your “hosts” variable? can you mask ip address partially and share? 192.x.y.z… etc

---

<div class="post-metadata">

**Author:** ![dhanasekaran1980](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@dhanasekaran1980](https://discuss.aerospike.com/u/dhanasekaran1980)\
**Post date:** [October 30, 2017, 4:25pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/5 "2017-10-30T16:25:27Z")

</div>

Gupta,

Below are the hosts

```
String aeroSpikeHost = "202.xxx.xxx.100,202.xxx.xxx.101,202.xxx.xxx.102"; // Arriving as parameter to this function
String aeroSpikePort = "3000,3000,3000"; // Arriving as parameter to this function
String hostsStr[] = aeroSpikeHost.split(",");
String portStr[] = aeroSpikePort.split(",");
Host[] hosts = new Host[hostsStr.length];
for (int i = 0; i < hosts.length; i++) {
    hosts[i] = new Host(hostsStr[i], Integer.parseInt(portStr[i]));
}

```

---

<div class="post-metadata">

**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\
**Post date:** [October 30, 2017, 4:32pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/6 "2017-10-30T16:32:49Z")

</div>

are you still getting the issue with maxSocketIldle at 55?

---

<div class="post-metadata">

**Author:** ![dhanasekaran1980](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@dhanasekaran1980](https://discuss.aerospike.com/u/dhanasekaran1980)\
**Post date:** [October 30, 2017, 4:41pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/7 "2017-10-30T16:41:14Z")

</div>

Gupta, Thanks for your quick reply. We are rolling out a release tonight with this change. Will reply once get tested.

---

<div class="post-metadata">

**Author:** ![dhanasekaran1980](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@dhanasekaran1980](https://discuss.aerospike.com/u/dhanasekaran1980)\
**Post date:** [October 30, 2017, 4:51pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/8 "2017-10-30T16:51:51Z")

</div>

Gupta,

Below is my service configuration. Can I retain proto-fd-max to 15000?

```
service {
    paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
    proto-fd-max 15000
    proto-fd-idle-max 60000
    transaction-pending-limit 0
    batch-max-requests 50000
}

```

---

<div class="post-metadata">

**Author:** ![dhanasekaran1980](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@dhanasekaran1980](https://discuss.aerospike.com/u/dhanasekaran1980)\
**Post date:** [October 30, 2017, 5:22pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/9 "2017-10-30T17:22:06Z")

</div>

Gupta,

Aerospike throws exception and not getting started saying parameter proto-fd-idle-max is invalid. Instead I set proto-fd-idle-ms 60000. But still Im getting com.aerospike.client.AerospikeException: Error Code 11: Server not available

Please suggest.

---

<div class="post-metadata">

**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\
**Post date:** [October 30, 2017, 9:52pm UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/10 "2017-10-30T21:52:06Z")

</div>

proto-fd-idle-ms is what I should have said. i mistyped. that is 60000 and maxSocketIdle should be 55. ie socket should be closed (connection reaping) by client before server closes it. setting socket idle 5 secs less than server side idle will ensure that.

Error Code 11 states: Server is not accepting requests. Occur during single node on a quick restart to join existing cluster.

---

<div class="post-metadata">

**Author:** ![dhanasekaran1980](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@dhanasekaran1980](https://discuss.aerospike.com/u/dhanasekaran1980)\
**Post date:** [October 31, 2017, 4:05am UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/11 "2017-10-31T04:05:40Z")

</div>

Gupta, This is occurring even after setting suggested properties while restarting a node on 3 node cluster. I am using community edition. Hence I don’t think I will be having Fast restart option. Please help.

---

<div class="post-metadata">

**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\
**Post date:** [October 31, 2017, 4:24am UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/12 "2017-10-31T04:24:09Z")

</div>

Once your client is connected to the cluster, one node going down should not matter to your connection to the other nodes. So I am not sure what is going on. Perhaps you can better explain the sequence of all events when you get the server error and someone else may chime in. With CE you don’t have access to Aerospike support otherwise they can troubleshoot using info collected from your cluster.

---

<div class="post-metadata">

**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\
**Post date:** [October 31, 2017, 5:26am UTC](https://discuss.aerospike.com/t/exception-on-node-restart/4724/13 "2017-10-31T05:26:59Z")

</div>

btw - what do you see for

`$ grep CLUSTER-SIZE /var/log/aerospike/aerospike.log`

Do you see CLUSTER-SIZE = 3?
