EAGAIN errors in Batch responses
After the server collects a response for a batch request, it must send that response to a waiting client.
In the non-batch path, Aerospike generally uses a thread per transaction response. However, in the batch subsystem, we collect a number of different responses all into a response queue. Each response queue has its own
batch-index-thread, which is sending responses as buffers fill up.
In Unix Sockets, with TCP, a send() call writes to an underlying kernel buffer. This buffer is called “wmem” and controlled with SysCtl. Some of the key variables are net.core.wmem_max and net.core.wmem ( and the related rmem parameters ). See man tcp for more information.
The buffer in question is not to be confused with an Ethernet driver’s transmit or receive queues. That involves the number of packets outstanding between the network device and the kernel, and has its own tuning rules.
With TCP, the userspace program calls the kernel with bytes to send. The kernel has only so much buffer, so copies some of the data into the kernel buffer. That buffer is only freed when the remote side sends a TCP ACK stating that the remote kernel ( not application ) has received the data.
Default sizes for these “rmem” buffers are between 32K and 64K per connection. Some guides propose increasing these values to 1M. This can impact servers with many open TCP connections, so a happy medium must be considered.
With Aerospike, it is necessary for one slow client to not impact other clients.
If the network response is small - smaller than the wmem buffer - then the thread transmitting on the server side will write, fill that buffer, and let the kernel do the repeated interrupt work of shipping the data. Only when that client, on that TCP connection, receives the data will the connection be used for another transaction.
If the network response is large - say, over 64K - then the send() call will either block, or copy some of the data and signal “EAGAIN” stating it only partially consumed the buffer.
With single-record responses in Aerospike, the thread will be responsible for filling the buffer, and retry until the buffer is clear. For a large response, this may take multiple network round trips. This consumes a transaction thread, but only one. If you have substantial number of large responses, and/or some slow clients, you may need to increase the number of these response threads (through
transaction-queues. However, other than the consumed transaction processing threads, there will be no impact between one slow client and other fast moving clients.
With multi-record responses (batch read), the situation is more complicated. Batch responses are usually much larger than wmax. Aerospike has a number of batch response threads (
batch-index-threads. A given batch request is assigned to a batch response thread, and collects the data in parallel with sending the response, when many keys are involved, in order to reduce latency.
In versions of Aerospike prior to 4.1, the batch response code would repeatedly attempt to send a 64K block, then could move on to the next 64K block response of another TCP connection. This would allow a slow client - which is not sending ACKs in a timely fashion - to increase latency for all the other connections assigned to that batch response thread. Increasing the number of batch response threads helps, but does not solve the problem.
In versions 4.1 and later, the batch response thread will attempt to send using a non-blocking send, then, on receiving an EWOULDBLOCK stating that the underlying buffer is full and the send was only partially consumed, would move to the next connection. This removes cases where there is a lack of threads, and a slow client will increase latency for other batch responses.
Two numbers bear monitoring: the abandoned logs, and the EAGAIN statistic (
An abandoned log line will look similar to this one:
abandoned batch from 184.108.40.206 with 23 transactions after 30000 ms.
Refer to the server log messages reference manual for further details.
As the different connections are tried repeatedly (using epoll()), the EAGAIN statistic (
batch_index_delay ) only shows that a particular client has a long response.
The abandoned log message (which will increase the
batch_index_error ) shows that a given batch send has timed out. Each batch request has a timeout, and when that time is reached (to be precise, twice that time or 30 seconds if not set), the server will not waste further time collecting responses and sending them back – instead, it terminates the connection and releases internal memory structures for the responses.
These kinds of “abandonment” can be caused by multiple reasons. Network issues may prevent sending the multiple packets and receiving the ACKs. The remote server itself (client in this case) may not have the processing power to receive the response, as the data must be delivered to the application on long responses (filling the receiver’s “rmem” buffer, notifying the application, having the data read, then frees memory in the rmem buffer, allowing the TCP window to re-open).
Diagnosing this kind of condition can be done with several tools.
A network packet analyser can show whether the TCP window reaches a fully closed state, and how long a closed window prevails for. If the period is very long, the client likely has some kind of a failure ( java garbage collection or application level bug ) preventing the server from sending. There may be excessive retransmissions, slowing network responses, which only are observed when the Aerospike server sends to the client — Aerospike may send a large number of back-to-back Ethernet packets, flooding driver packet buffers or switch buffers.
A quick diagnosis can be done with
netstat to check if any network Send-Q or Recv-Q queue backup is occuring for a particular IP address. In case of client being overloaded (ie: high CPU) you would see high Send-Q on server for the client IP, and high Recv-Q on client machine in
netstat. This could indicate that client hosts are running low on resources and there is a bottleneck processing the network receive queue on the client. The next step would be to check CPU usage on the client.
Server: netstat -pant|egrep 'ESTA|Send' (No info could be read for "-p": geteuid()=1000 but you should be root.) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 305016 220.127.116.11:3000 18.104.22.168:49028 ESTABLISHED - Client: netstat -pant|egrep 'ESTA|Send' (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 263700 0 22.214.171.124:49028 126.96.36.199:3000 ESTABLISHED 5402/java
A simple solution is to increase the BatchPolicy timeout.
EAGAIN BATCH TCP wmem rmem wmem_max rmem_max