Least Available Percentage (Avail Pct) decreasing

Operations Planning
1

Hello,

I can not figure out how to fix least available% which is getting lower.

I have tried coldstart with no effect and after that I added one more file to storage-engine, disk capacity increased but least avail remains the same.

99% of my data have ttl=0

Any suggestions? Should I increase “filesize” in config?

I am using Aerospike Community 3.5.12

namespace config:

namespace dmp {
    replication-factor 2
    memory-size 12G
    default-ttl 0

    ldt-enabled true

    storage-engine device {
        file /data/aerospike/data/dmp.dat
        file /data/aerospike/data/dmp1.dat
        file /data/aerospike/data/dmp2.dat
        filesize 20G
        data-in-memory false
    }
}
asmonitor -e "info Namespace"

Enter help for commands

1 hosts in cluster: 127.0.0.1:3000
Total (unique) objects in cluster for dmp : 50,276,574
Note: Total (unique) objects is an under estimate if migrations are in progress.

 === NAMESPACE ===
ip/namespace                    Avail   Evicted       Master     Repl     Stop      Used   Used     Used   Used    hwm   hwm
                                  Pct   Objects      Objects   Factor   Writes      Disk   Disk      Mem    Mem   Disk   Mem
                                    .         .            .        .        .         .      %        .      %      .     .
127.0.0.1/dmp                      14         0   50,276,574        1    false   15.58 G     26   3.66 G     31     50    60
No. of rows: 1
asinfo -v "namespace/dmp"

type=device;objects=50268881;sub-objects=0;master-objects=50268881;master-sub-objects=0;prole-objects=0;prole-sub-objects=0;expired-objects=12787;evicted-objects=0;set-deleted-objects=0;set-evicted-objects=0;nsup-cycle-duration=16;nsup-cycle-sleep-pct=0;used-bytes-memory=3933280284;data-used-bytes-memory=0;index-used-bytes-memory=3217208384;sindex-used-bytes-memory=716071900;free-pct-memory=69;max-void-time=174645451;non-expirable-objects=50230231;current-time=174472655;stop-writes=false;hwm-breached=false;available-bin-names=32663;ldt-reads=0;ldt-read-success=0;ldt-deletes=0;ldt-delete-success=0;ldt-writes=0;ldt-write-success=0;ldt-updates=0;ldt-gc-io=0;ldt-gc-cnt=0;ldt-randomizer-retry=0;ldt-errors=0;ldt-err-toprec-notfound=0;ldt-err-item-notfound=0;ldt-err-internal=0;ldt-err-unique-key-violation=0;ldt-err-insert-fail=0;ldt-err-delete-fail=0;ldt-err-search-fail=0;ldt-err-version-mismatch=0;ldt-err-capacity-exceeded=0;ldt-err-param=0;ldt-err-op-bintype-mismatch=0;ldt-err-too-many-open-subrec=0;ldt-err-subrec-not-found=0;ldt-err-bin-does-not-exist=0;ldt-err-bin-exits=0;ldt-err-bin-damaged=0;ldt-err-toprec-internal=0;ldt-err-subrec-internal=0;ldt-err-filer=0;ldt-err-key=0;ldt-err-createspec=0;ldt-err-usermodule=0;ldt-err-input-too-large=0;ldt-err-ldt-not-enabled=0;ldt-err-unknown=0;used-bytes-disk=16724714368;free-pct-disk=74;available_pct=14;cache-read-pct=72;memory-size=12884901888;high-water-disk-pct=50;high-water-memory-pct=60;evict-tenths-pct=5;stop-writes-pct=90;cold-start-evict-ttl=4294967295;repl-factor=1;default-ttl=0;max-ttl=0;conflict-resolution-policy=generation;allow_versions=false;single-bin=false;ldt-enabled=true;ldt-page-size=8192;enable-xdr=false;sets-enable-xdr=true;ns-forward-xdr-writes=false;allow-nonxdr-writes=true;allow-xdr-writes=true;disallow-null-setname=false;total-bytes-memory=12884901888;read-consistency-level-override=off;write-commit-level-override=off;total-bytes-disk=64424509440;defrag-lwm-pct=50;defrag-queue-min=0;defrag-sleep=1000;defrag-startup-minimum=10;flush-max-ms=1000;fsync-max-sec=0;write-smoothing-period=0;max-write-cache=67108864;min-avail-pct=5;post-write-queue=256;data-in-memory=false;file=/data/aerospike/data/dmp.dat;file=/data/aerospike/data/dmp1.dat;file=/data/aerospike/data/dmp2.dat;filesize=21474836480;writethreads=1;writecache=67108864;obj-size-hist-max=100
2

And here is output from aerospike.log

Jul 13 2015 08:56:02 GMT: INFO (drv_ssd): (drv_ssd.c::2536) device /data/aerospike/data/dmp2.dat: used 243764992, contig-free 20049M (20049 wblocks), swb-free 1, n-w 0, w-q 0 w-tot 671 (0.0/s), defrag-q 0 defrag-tot 242 (0.0/s)
Jul 13 2015 08:56:02 GMT: INFO (drv_ssd): (drv_ssd.c::2536) device /data/aerospike/data/dmp.dat: used 11133398784, contig-free 2968M (2968 wblocks), swb-free 1, n-w 0, w-q 0 w-tot 724 (0.0/s), defrag-q 0 defrag-tot 598 (0.0/s)
Jul 13 2015 08:56:02 GMT: INFO (drv_ssd): (drv_ssd.c::2536) device /data/aerospike/data/dmp1.dat: used 5369252224, contig-free 12156M (12156 wblocks), swb-free 1, n-w 0, w-q 0 w-tot 699 (0.0/s), defrag-q 0 defrag-tot 592 (0.0/s)
3

Hi,

You can increase defrag-lwm-pct to see if Avail Pct gets better, increasing defrag-lwm-pct makes defrag process aggressive by making more blocks available for defragmentation. Here is the link discussing about similar scenario, Recovering from Available Percent Zero

4

set defrag-lwm-pct to 60, least available is 48% now. Thank you!

1 Like