Running a 3.8.x (or above) version of XDR may generate the following errors in the logs. Additional behaviors could be an increase in records outstanding, a zero (or low number) throughput accompanied with relogs and retransmits.
WARNING (xdr): (as_pipe.c:247) Buffer limit is XXXXXXXXX, should be at least 5242880 for async pipelining. Please set /proc/sys/net/core/wmem_max accordingly. WARNING (xdr): (as_pipe.c:247) Buffer limit is 3000000, should be at least 15728640 for async pipelining. Please set /proc/sys/net/core/rmem_max accordingly. DEBUG (xdr): (xdr.c:3669) Faced a client driver level error. Code -6: Socket 190 read closed by peer at /work/source/modules/ee/xdr/src/as_event_ev.c:313. Relogging digest.
Newer version of XDR now run in the ASD process and also uses async pipelining for writes replication. XDR pipelining will aim to put a lot of records on a single connection as compared to the pre 3.8 XDR. For this purpose the tcp buffers should be increased. If the buffer limit kernel settings (wmem_max and rmem_max) are not increased, the impact of the debug message that we saw (about setting the Buffer limit setting on the cluster) is that the remote DC will close client connections instead of receiving a server response. If the buffer limit is not configured properly, the remote DC will refuse the client (XDR) connection and close it (instead of sending a server response).
Note that buffer limit kernel settings missing at the source DC would simply cause XDR to ship at a slower rate whether a missing configuration of those settings at the destination DC would actually prevent XDR from even establishing connections properly.
The buffer limit needs to be configured properly on all cluster nodes in both DCs, local and remote.
The kernel settings for
rmem_max need to at least be set to the recommeded values.
wmem_max 5242880 rmem_max 15728640
Note: These values are set by the default installation script
init.d aerospike (startup script) if on a standard installation of Aerospike.
If using an embedded version of aerospike the kernel settings can be set through sysctl.
Sysctl commands to manually set buffers
Modify kernel setting prior to restarting asd.
sysctl -w net.core.rmem_max=15728640 sysctl -w net.core.wmem_max=5242880
and edit file to make change persist: /etc/sysctl.conf
XDR buffer rmem wmem pipeline