XDR showing buffer limit error on startup


#1

Problem Description

Running a 3.8.x (or above) version of XDR may generate the following errors in the logs. Additional behaviors could be an increase in records outstanding, a zero (or low number) throughput accompanied with relogs and retransmits.

WARNING (xdr): (as_pipe.c:247) Buffer limit is XXXXXXXXX, should be at least 5242880 for async pipelining. Please set /proc/sys/net/core/wmem_max accordingly.
WARNING (xdr): (as_pipe.c:247) Buffer limit is 3000000, should be at least 15728640 for async pipelining. Please set /proc/sys/net/core/rmem_max accordingly.
DEBUG (xdr): (xdr.c:3669) Faced a client driver level error. Code -6: Socket 190 read closed by peer at /work/source/modules/ee/xdr/src/as_event_ev.c:313. Relogging digest.

Explanation

Newer version of XDR now run in the ASD process and also uses async pipelining for writes replication. XDR pipelining will aim to put a lot of records on a single connection as compared to the pre 3.8 XDR. For this purpose the tcp buffers should be increased. If the buffer limit kernel settings (wmem_max and rmem_max) are not increased, the impact of the debug message that we saw (about setting the Buffer limit setting on the cluster) is that the remote DC will close client connections instead of receiving a server response. If the buffer limit is not configured properly, the remote DC will refuse the client (XDR) connection and close it (instead of sending a server response).

Note that buffer limit kernel settings missing at the source DC would simply cause XDR to ship at a slower rate whether a missing configuration of those settings at the destination DC would actually prevent XDR from even establishing connections properly.

Solution

The buffer limit needs to be configured properly on all cluster nodes in both DCs, local and remote.

The kernel settings for wmem_max and rmem_max need to at least be set to the recommeded values.

wmem_max 5242880
rmem_max 15728640

Note: These values are set by the default installation script init.d aerospike (startup script) if on a standard installation of Aerospike.

If using an embedded version of aerospike the kernel settings can be set through sysctl.

Sysctl commands to manually set buffers

Modify kernel setting prior to restarting asd.

sysctl -w net.core.rmem_max=15728640
sysctl -w net.core.wmem_max=5242880

and edit file to make change persist: /etc/sysctl.conf

net.core.rmem_max=15728640
net.core.wmem_max=5242880

Keywords

XDR buffer rmem wmem pipeline

Timestamp

6/28/16