FAQ Tuning async Java Client connections

Knowledge Base Client APIs
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

FAQ Tuning the Async Java Client connections

Why use the Async Client:

  • If the application is deployed within an asynchronous framework.

  • The Async API provides tighter controls through event loop and single thread.

In general, it is preferred to use the Synchronous API as it is much easier to tune and debug and the potential performance gains for general workloads and use cases may not be noticeable.

Java client version 4.3.0 and above

AerospikeClient asynchronous methods contain an EventLoop argument that allows the user to specify which event loop thread should process the command. Refer to Async API documentation reference for details.

  • The number of event loops for an EventLoops instance should approximate the number of CPU cores on your machine that are reserved for AerospikeClient processing.

maxCommandsInProgress

Used to limit the number of concurrent commands allowed on the event loop.

The upper bound for open connections is:

(Number of event loops) x (maxCommandsInProcess) x (Number of nodes)

Let’s assume N = (Number of event loops) x (maxCommandsInProgress)

The upper bound is achieved if a sequence of N commands went to a single node, thus filling up that node’s connection pool. Then, the next sequence of N commands went to a different node and so on, until all node’s connection pools were filled up.

The number of open connections can exceed N because connections are cached in connection pools (one pool per node per Event loop). maxCommandsInProcess limits the number of concurrently executing commands/connections, but there are extra connections sitting in each node’s connection pools. These extra connections in each node are useful because command distribution across nodes is never exactly evenly distributed. If commands are suddenly directed more to one node, those commands will use those extra connections instead of creating new connections. Every connection pool has these extra connections to account for uneven node distribution.

maxSocketIdle

Additionally, the total amount of open sockets correlates to the amount of sockets consumed at peak usage. Connection pools are trimmed when connections in the pool sit unused for longer than ClientPolicy.maxSocketIdle or a timeout/network error occurred on a connection. The new stack-based connection pool implementation in version 4.3.1 (Internal Jira CLIENT-1059) enforces maxSocketIdle more strictly than in the previous versions.

maxConnsPerNode

To limit the number of connections, consider using maxConnsPerNode, but note that if the max limit is hit, the client will throw NO_MORE_CONNECTIONS exception.

Java client versions 4.2.3 and earlier

Refer to Incompatability Changes for more details.

Async client tuning:

The following knobs can be used to tune the async Java Client connections: asyncMaxCommandAction, asyncMaxCommands, asyncSelectorThreads, asyncSelectorTimeout, timeoutDelay

In more details:

asyncMaxCommandAction (DEFAULT: BLOCK)

asyncMaxCommandAction uses 3 states to handle cases when the asynchronous maximum number of concurrent connections have been reached:

  • ACCEPT: Accept and process command. This implies the user is responsible for throttling asynchronous load. You are not limited when choosing Accept.
  • BLOCK: Block until a previous command completes. (Default value)
  • REJECT: Reject database command.

asyncMaxCommands (DEFAULT: 200)

asyncMaxCommands has a default value of 200 commands (tested on enterprise class machine with 8 cores). This value is the maximum number of concurrent asynchronous commands that are active at any point in time. Concurrent commands can be used to estimate concurrent connections. The number of concurrent open connections is limited by:

max open connections = asyncMaxCommands * Number of Cluster Nodes

The actual number of open connections consumed depends on how balanced the commands are between nodes and if asyncMaxCommandAction is set to ACCEPT.

The maximum open connections should not exceed the total socket file descriptors available on the client machine. The socket file descriptors available can be determined by the following command: ulimit -n

asyncSelectorThreads (DEFAULT: 1)

asyncSelectorThreads is the number of event loops configured in the client.(Java Library calls Selector what C client calls an events loop) Number of selector threads used to process asynchronous network events. The default is a single threaded network handler. Some applications may benefit from increasing this value to the number of unused cores on a machine running other Core dependent services… There is no benefit to having more threads than you have cores.

ayncSelectorTimeout (DEFAULT: 0)

ayncSelectorTimeout has a default of zero (milliseconds) with no timeouts of selector. Default may have to change. Every time you put a new command on the queue, the selector wakes up. If using async client sporadically, then you should modify the asyncSelectorTimeout to limit Wait state (if no commands sent).

For example: if transactions of 5 ms, the Selector timeout should also be at 5ms.

How timeouts are implemented in java async.
  1. run Selector thread
  2. Add new command to timeout queue
  3. Check for timeout
  4. Enter Selector wait state until it fires some results
  5. Process Socket reads and Writes.

In Wait state we do not check for timeout. It is not an independent thread.

ayncSelectorTimeout should be average timeout for your transactions. The exception would be if you are always pumping data through client all the time. Than no risk to go into a Wait State for too long. Low Selector Timeout may increase CPU usage as you are checking for timeouts more often.

timeoutDelay (DEFAULT: 0 (no delay, connection closed on timeout))

It may be worth also tuning timeoutDelay if you have aggressive timeout and you are closing a lot of connections. This may result in an open and close loop with lots of Socket in Close Wait state.(There is a cost to opening Socket connections). The timeoutDelay is a delay in milliseconds after a transaction timeout before closing a socket in async mode only. Async transactions only retry on invalid connections in the connection pool which is not bounded. When a transaction is stopped prematurely, the socket must be closed and not placed back on the pool. This is done to prevent unread socket data from corrupting the next transaction that would use that socket. This field delays the closing of the socket to give the transaction more time to complete in the hope that the socket can be reused. This is helpful when timeouts are aggressive and a certain percentage of timeouts is expected. The user is still notified of the timeout in async mode at the original timeout value. The transaction’s async timer is then reset to this delay and the transaction is allowed to continue. If the transactions succeeds within the delay, then the socket is placed back on the pool and the transaction response is discarded. Otherwise, the socket must be closed. This field is ignored in sync mode because control must be returned back to user on timeout and there is no currently available thread pool to process the delay.

Recommendation - Iterative method

Tuning async client requires running a benchmark on your system. It’s usually best to iterate starting with default values and incrementing while monitoring your service. asyncSelectorThreads should not exceed number of free CPU cores on your client. One should take into consideration any other applications running on the client machine that may also be using dedicated CPU Cores when adjusting this number. Trial and error to minimize resource consumption and maximize Throughput.

Reference:

Keywords

asyncMaxCommandAction, asyncMaxCommands , asyncSelectorThreads , ayncSelectorTimeout

Timestamp

October 2019