Getting "Cluster has been closed" Error on a Client

Client Libraries Java Client
1

Hey,

I am getting Cluster has been closed error on one of the client boxes and when I restart the box the error goes away.

Please note that the error comes only on one of the client boxes, not all, and all client boxes have equally distributed command invocation(s).

I am doing Async Commands (Read + Write + UDF).

Client Library Version: Java: 4.0.6

Can you please suggest how to resolve this error?

2

Further debugging inside Aerospike Codebase,

Found that: NioCommand.java class

        if (eventState.pending++ == -1) {
			eventState.pending = -1;
			eventState.errors++;
			state = AsyncCommand.COMPLETE;		
			notifyFailure(new AerospikeException("Cluster has been closed"));
			return;
		}

and I can see that eventState.pending-- is getting called at two places:

  1. At run() method of NioCommand class, when command deadline is exceeded (i.e., timeout)
  2. At close() method of NioCommand class

Client Library Version: Java: 4.0.6

3

This error happens when an async command has been initiated after AerospikeClient.close() was called. Did your code call AerospikeClient.close() around the same time you received the error?

4

No, Brian.

I am calling AerospikeClient.close() in shutdown hook only. (shutdown hook is not called as I was able to see other process life-cycles working fine and Cluster has been closed error being thrown continuously).

Can you please mention what other reasons can be there to do eventState.pending’s value to be -2?

Also, please guide me towards what other logs/info can help debug this scenario.

5

eventState.pending should never be -2. I suggest trying your code with the latest java client to see if that fixes the problem.

6

Ok, will try new version, and keep this thread updated. Thanks