Disgestlog partition out of space


#1

Synopsis:

The disk/partition on which the XDR digestlog is configured is running out of space.

Explanation

Confirm the value of digestlog-path in aerospike.conf file:

digestlog-path /etc/aerospike/digestlog 100000000000

Digestlog will not grow over this configured limit. Digest log uses a circular buffer ring and will start overwriting older data with latest digest entries.

Digest log’s actual size can be verified with the “-s” option of ls

ls -alsh /etc/aerospike/digestlog
9.4G -rw------- 1 root root 9.4G Oct 23 14:48 /etc/aerospike/digestlog

The first size column is the effective space taken on disk while the second size column is the max space of that file.

or

/etc/aerospike$ du -sh digestlog
9.4G digestlog

Solution

The Digest log can be moved to a different partition with more space.

Steps are as follows.

Versions prior to 3.8

  1. stop XDR. sudo /etc/init.d/aerospike_xdr stop
  2. Move existing Digestlog to new location. (Or delete if that is an option)
  3. Reconfigure XDR disgeslog file location path (digestlog-path ) in aerospike.conf file.
  4. restart XDR sudo /etc/init.d/aerospike_xdr resume

Versions 3.8 and above require a restart of the aerospike service

If you see the following line in your XDR log file, then your dlog has most certainly wrapped around (assuming your xdr logging-level is set to info) :

"Windowshipper: Digest log got wrapped. Continuing from the current start marker"

The digestlog statistics can be referred to using this command:

asinfo -v ‘statistics’ -l | grep dlog

Additional info at: http://www.aerospike.com/docs/operations/manage/xdr/

Keywords

DIGESTLOG XDR PARTITION SPACE

Timestamp

1/11/2018


XDR ran out of queue space
Ran out of queue space... XDR cannot keep up with write