However, in our code, asyncPolicy.asyncMaxCommands was explicitly set to 200.
From inspecting the C# client code, we think we found a possible explanation as to why it has happened.
It looks like the AsyncClient doesn’t actually enforce the asyncMaxCommands in the async client policy.
In AsyncCommand.ExecuteCommand():
conn = node.GetAsyncConnection();
if (conn == null)
{
conn = new AsyncConnection(node.address, cluster);
. . .
}
AsyncNode.GetAsyncConnection will return null when there are no more available connections, i.e. asyncConnQueue.TryTake fails.
So in case of a high burst\spike of commands that exceeds asyncMaxCommands, new connections would be created regardless of the number of open connections\commands already used.
asyncMaxCommands is only enforced during command completion (AsyncCommand.Finish) when the connection is returned to AsyncNode (PutAsyncConnection).
BTW, from inspecting the java client (which is similar to the C# client), it appears the same behavior exists there too.
To circumvent this issue, we are now throttling all async calls to the AsyncClien.
However, as opposed to asyncMaxCommands, it’s not possible to throttle on a per node basis - only on the global concurrent open async commands for the entire client.
AsyncClient does enforce asyncMaxCommands when it retrieves a SocketAsyncEventArgs from a fixed sized pool. This is the first method called in AsyncCommand.Execute().
public SocketAsyncEventArgs GetEventArgs()
{
if (block)
{
// Use blocking retrieve from queue.
// If queue is empty, wait till an item is available.
return argsQueue.Take();
}
// Use non-blocking retrieve from queue.
SocketAsyncEventArgs args;
if (argsQueue.TryTake(out args))
{
return args;
}
// Queue is empty. Reject command.
throw new AerospikeException.CommandRejected();
}
The connections are created on the fly because the connection pool is initially empty. Since there is one connection per command and the commands are limited, then the connections are limited too.
I’m still trying to figure out then how we’ve reached to 15,000 client connections per Aerospike node with asyncMaxCommands=200
I looked more into the client code, and have some followup questions regarding this:
Since argsQueue is defined in the AsyncCluster and is bounded to asyncMaxCommands, wouldn’t this mean that the asyncMaxCommands is enforced for the entire cluster and not per node as mentioned in the documentation (max open connections = asyncMaxCommands * number of nodes in cluster)?
In AsyncCommand.RetryOnInit, there is no actual sleep between retries:
// Disable sleep between retries. Sleeping in asynchronous mode makes no sense.
//if (watch != null && (watch.ElapsedMilliseconds + policy.sleepBetweenRetries) > timeout)
if (watch != null && watch.ElapsedMilliseconds > timeout)
{
// Might as well stop here because the transaction will
// timeout after sleep completed.
return FailOnNetworkInit();
}
What was the reason for disabling sleepBetweenRetries?
Without sleep between retries, and with maxRetries=3, wouldn’t there be 4 attempts to connect to the same node (I assume it takes time before a node is marked as inactive or disconnected)?
Does this mean that for Async Client it is better to avoid having retries at all?
1) asyncMaxCommands is enforced for the entire cluster. That documentation blurb is also correct, but could be misinterpreted.
If a large block of commands were exlusively sent to the same node, then that large block would ultimately result in that node’s connection pool being filled to a limit of asyncMaxCommands.
Then, another large block of commands is exlusively sent to the another node. That node’s connection pool would also fill up to a limit of asyncMaxCommands. Repeat this process for all nodes in the cluster.
Since connections will not be dropped from the pools, the total number of open connections (including connections in the pools) is indeed limited by “asyncMaxCommands * nodeCount”.
In most cases, node distribution is more even, so the total number of open connections will be less than the theorectical max.
My original statement in this thread should have pointed out that even though each active command contains one connection, there will be more connections used than commands because of the extra connections stored in the connection pools.
2) sleepBetweenRetries makes sense for sync commands because the sleep is performed in parallel by multiple threads. The retry sleep time when a node goes down would be sleepBetweenRetries even when running 100 concurrent threads.
sleepBetweenRetries does not makes sense for async commands because the sleep is performed in series for each async event loop thread. If a node goes down, the first async command to that node would sleep for sleepBetweenRetries, then the next command would sleep and so on in sequence. This would result in huge client downtimes even when the node comes back up because existing async command sockets will not be reopened until they fail and sleep.
Yes, it is better to avoid internal client retries completely in async mode. If a command really needs to be retried, then it can be placed on an external retry queue that is controlled and processed by the user.