XDR not shipping after upgrading to 3.8.1 XDR in ASD


#1

XDR not shipping from one or multiple nodes after upgrading to new 3.8.1 version.

Problem Description

After upgrading to 3.8.1 there could be a possibilty that XDR will show zero throughput on one or many nodes. And you see this message in the log file.

Apr 19 2016 09:47:40 GMT-0400: WARNING (cf:socket): (socket.c:249) bind: socket in use, waiting (port:3004)

Explanation

This is due to the fact that while upgrading, the old XDR process wasn’t stopped and both old and new xdr are trying to use the same digest log file. The above warning about “socket in use” is because of the old XDR port still defined in the configuration.

Even if the configuration is updated, digestlog will still be corrupted if the old XDR process is not stopped before the new ASD is installed.

Solution

To recover from either situations:

Stop the old XDR . Delete the digest log file and restart ASD.

sudo service aerospike_xdr stop
sudo service aerospike restart

For versions 3.8.3 and above, you can issue a XDR restart by ignoring the pending records in the digestlog to mitigate the issue:

asinfo -v 'xdr-command:skip-outstanding'

Notes

Please follow the rolling upgrade to XDR in ASD doc diligently while upgrading to the new XDR. http://www.aerospike.com/docs/operations/upgrade/xdr_to_3_8 http://www.aerospike.com/docs/operations/upgrade/xdr_to_3_8/release_details_3_8

Keywords

XDR, Upgrade, 3.8.1, xdr_in_asd.

Timestamp

4/21/16