The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
FAQ - Understanding defragmentation after a cold-restart
Detail
A cold-start will trigger a linear scan of all disk devices. During this scan, the objects will be in random order in terms of their potential different versions. If Durable Deletes are not used, it is possible that a current version of an object from a block that was previously defragmented and not overwritten becomes active after a cold-start. Refer to the Cold-Start Resurrecting Deleted Records article for this specific behavior. That object version and its block will once again be eligible for another round of defragmentation. The identical copies of that object will be ignored as the scan progresses. The scan picked up the “wrong” defragged version of an object (from previously defragged block) and lots of blocks that were free are no longer free. In other words after the Cold start, Aerospike went into a state of “undefrag” and needs to re-defragment previously defragmented blocks.
Defrag Init state (Undefrag) in the logs
After a cold-start, the server logs will show an init defrag profile that will indicate the amount of blocks that will be put in the defrag-q for defragmentation:
Jul 17 2018 21:11:41 GMT: INFO (drv_ssd): (drv_ssd.c:1058) {NAMESPACE} loading free & defrag queues
Jul 17 2018 21:11:41 GMT: INFO (drv_ssd): (drv_ssd.c:992) /dev/sdf1 init defrag profile: 505277,250898,187527,159354,152533,121549,108080,95366,92442,73832,64653,57008,55355,43187,38069,33028,31671,25082,22009,19204,18168,14360,12561,10950,9408,9065,7219,6473,5432,5303,4205,3678,3344,3098,2578,2290,2089,2056,1625,1482,1402,1404,1211,1162,1144,1230,1055,1077,1030,1077,1226,1162,1175,1355,1604,1649,1728,1981,2466,2590,2880,3444,4160,4642,5276,6250,7796,8395,9842,11391,15108,15646,18295,21308,24841
Jul 17 2018 21:11:41 GMT: INFO (drv_ssd): (drv_ssd.c:992) /dev/sdb1 init defrag profile: 516112,258270,192197,163248,157115,125177,110622,97590,95127,75896,66332,58410,55759,44507,38963,33617,32618,25628,22241,19724,18655,14652,12923,11316,9795,9328,7327,6428,5652,5410,4387,3792,3333,3229,2581,2323,2113,2088,1775,1567,1431,1487,1210,1201,1134,1230,1102,1113,1085,1083,1248,1215,1255,1406,1668,1536,1779,2034,2522,2592,2995,3408,4536,4696,5434,6481,8293,8676,10147,11674,15114,16071,18648,21798,25490
Jul 17 2018 21:11:41 GMT: INFO (drv_ssd): (drv_ssd.c:992) /dev/sde1 init defrag profile: 516382,257575,193379,164387,155904,125577,110372,97910,94582,75702,66563,58539,56379,44210,39020,34229,32613,25845,22309,19870,18764,14848,13062,11193,9704,9325,7462,6453,5660,5562,4148,4013,3398,3344,2558,2363,2164,2027,1787,1542,1495,1479,1266,1189,1152,1144,1110,1062,1097,1096,1250,1241,1218,1334,1632,1661,1889,2024,2575,2652,3095,3502,4400,4708,5507,6273,8208,8633,10221,11890,15305,16431,18749,21721,25490
Jul 17 2018 21:11:41 GMT: INFO (drv_ssd): (drv_ssd.c:992) /dev/sdd1 init defrag profile: 515834,258568,193454,163316,156345,125001,110788,98249,94316,75818,66413,58545,56132,44537,38828,34210,32704,25543,22400,19559,18722,14675,12897,11184,9749,9338,7206,6410,5675,5400,4408,3789,3352,3256,2593,2298,2148,2151,1681,1595,1503,1547,1289,1195,1169,1199,1106,1051,1047,1138,1227,1157,1304,1387,1665,1668,1733,2100,2633,2686,3002,3448,4582,4683,5575,6460,8180,8664,9943,11748,15384,16342,18727,21662,25243
Jul 17 2018 21:11:41 GMT: INFO (drv_ssd): (drv_ssd.c:992) /dev/sdc1 init defrag profile: 515913,257553,192531,163886,155773,124710,110501,98270,94732,75655,66593,58331,55882,44374,38867,33941,32741,25563,22322,19692,18618,14748,12773,11149,9873,9425,7370,6548,5706,5369,4378,3785,3300,3256,2559,2393,2013,2091,1729,1549,1440,1495,1269,1224,1165,1288,1107,1151,1096,1057,1221,1164,1220,1384,1628,1642,1790,2030,2529,2643,3090,3458,4376,4595,5493,6456,7989,8710,10251,11659,15276,16185,18760,21893,25549
Jul 17 2018 21:11:41 GMT: INFO (drv_ssd): (drv_ssd.c:992) /dev/sdg1 init defrag profile: 503575,251995,188078,159797,152352,121744,108145,95570,92557,73629,64798,57267,54646,43501,38097,33540,31911,25106,21979,19119,18162,14432,12460,10918,9436,9220,7295,6388,5547,5242,4217,3722,3253,3179,2581,2248,2096,1998,1709,1529,1380,1497,1190,1241,1149,1200,1083,1055,1066,1024,1216,1261,1221,1260,1540,1550,1753,1942,2505,2644,2865,3298,4297,4604,5257,6195,8044,8395,10106,11372,15042,15837,18203,21421,24974
Jul 17 2018 21:11:42 GMT: INFO (drv_ssd): (drv_ssd.c:1082) /dev/sdb1 init wblock free-q 2996751, defrag-q 2510619
Jul 17 2018 21:11:42 GMT: INFO (drv_ssd): (drv_ssd.c:1082) /dev/sdc1 init wblock free-q 2998308, defrag-q 2509745
Jul 17 2018 21:11:42 GMT: INFO (drv_ssd): (drv_ssd.c:1082) /dev/sdd1 init wblock free-q 2995841, defrag-q 2512534
Jul 17 2018 21:11:42 GMT: INFO (drv_ssd): (drv_ssd.c:1082) /dev/sde1 init wblock free-q 2993900, defrag-q 2514423
Jul 17 2018 21:11:42 GMT: INFO (drv_ssd): (drv_ssd.c:1082) /dev/sdf1 init wblock free-q 3142237, defrag-q 2449510
Jul 17 2018 21:11:42 GMT: INFO (drv_ssd): (drv_ssd.c:1082) /dev/sdg1 init wblock free-q 3140579, defrag-q 2451725
The init defrag profile table above will depend on the defrag-lwm-pct setting and will have an equal number of buckets.
In the above example the comma separated list will have 75 entries, matching the defrag-lwm-pct setting.
The first bucket will have the wblocks count that are 0 to 1% full.
The second bucket will have the wblocks count that are 1 to 2% full.
…
The 75th bucket will have the wblocks count that are 74 to 75% full.
At cold-start, the defragmentation process will start with the lower buckets (most depleted blocks first) up to the higher ones in order to generate as many free blocks as possible for as few blocks filled and written as possible to accommodate incoming migrations and application traffic.
Notes
Setting defrag-lwm-pct higher than the recommended value of 50 percent should be done carefully. Increasing this value will result in an exponential amplification of writes. Refer to the article explaining the reasons behind the 50% default value for further details.
Keywords
DEFRAG COLD-START UNDEFRAG DEFRAGMENTATION REDEFRAG
Timestamp
07/20/2018