XDR not shipping from one or multiple nodes after upgrading to new 3.8.1 version.
After upgrading to 3.8.1 there could be a possibilty that XDR will show zero throughput on one or many nodes. And you see this message in the log file.
Apr 19 2016 09:47:40 GMT-0400: WARNING (cf:socket): (socket.c:249) bind: socket in use, waiting (port:3004)
This is due to the fact that while upgrading, the old XDR process wasn’t stopped and both old and new xdr are trying to use the same digest log file. The above warning about “socket in use” is because of the old XDR port still defined in the configuration.
Even if the configuration is updated, digestlog will still be corrupted if the old XDR process is not stopped before the new ASD is installed.
To recover from either situations:
Stop the old XDR . Delete the digest log file and restart ASD.
sudo service aerospike_xdr stop sudo service aerospike restart
For versions 3.8.3 and above, you can issue a XDR restart by ignoring the pending records in the digestlog to mitigate the issue:
asinfo -v 'xdr-command:skip-outstanding'
Please follow the rolling upgrade to XDR in ASD doc diligently while upgrading to the new XDR. http://www.aerospike.com/docs/operations/upgrade/xdr_to_3_8 http://www.aerospike.com/docs/operations/upgrade/xdr_to_3_8/release_details_3_8
XDR, Upgrade, 3.8.1, xdr_in_asd.