Asbackup fails with error bad file descriptor
When running asbackup to perform a backup, the following error is thrown:
asbackup --file-limit 1024 MiB --compact --namespace default --directory /aero-backup/default/10.0.0.50 --node-list 10.0.0.50:3000 --priority 0 --nice 10240 Error while running node scan for BB95E01C0342712 - code -1: Bad file descriptor at src/main/aerospike/as_socket.c:358
The aerospike servers have a timeout parameter, which may time the connection out. Asbackup utility sets the client timeout to 10 seconds as well. This may result in the conneciton being closed down. When asbackup attempts to read from that conneciton, it fails as it has been terminated.
There can be multiple causes for this, all being related to the parameters used, as follows:
- if the disks are too slow, used in conjunction with a large file-limit, this may cause asbackup to fail. Asbackup commits large flushes to disk at once. If the file-limit is increased from 250MB, this causes a larger flush to disk, and may result in taking too long. Try lowering the file-limit paramter if your disks are not managing to flush data fast enough.
- alternatively, you may experience a timeout if the network is too slow or the backup server cannot read from the socket fast enough. In such a case, you can either adjust the ‘nice’ level to allow for higher bandwidth, or limit the number of nodes on which asbackup runs at once in parallel using the ‘parallel’ parameter to a lower value.