The disk/partition on which the XDR digestlog is configured is running out of space.
Confirm the value of digestlog-path in aerospike.conf file:
digestlog-path /etc/aerospike/digestlog 100000000000
Digestlog will not grow over this configured limit. Digest log uses a circular buffer ring and will start overwriting older data with latest digest entries.
Digest log’s actual size can be verified with the “-s” option of ls
ls -alsh /etc/aerospike/digestlog 9.4G -rw------- 1 root root 9.4G Oct 23 14:48 /etc/aerospike/digestlog
The first size column is the effective space taken on disk while the second size column is the max space of that file.
/etc/aerospike$ du -sh digestlog 9.4G digestlog
The Digest log can be moved to a different partition with more space.
Steps are as follows.
Versions prior to 3.8
- stop XDR.
sudo /etc/init.d/aerospike_xdr stop
- Move existing Digestlog to new location. (Or delete if that is an option)
- Reconfigure XDR disgeslog file location path (digestlog-path ) in aerospike.conf file.
- restart XDR
sudo /etc/init.d/aerospike_xdr resume
Versions 3.8 and above require a restart of the aerospike service
If you see the following line in your XDR log file, then your dlog has most certainly wrapped around (assuming your xdr logging-level is set to info) :
"Windowshipper: Digest log got wrapped. Continuing from the current start marker"
The digestlog statistics can be referred to using this command:
asinfo -v ‘statistics’ -l | grep dlog
Additional info at: http://www.aerospike.com/docs/operations/manage/xdr/
DIGESTLOG XDR PARTITION SPACE