Write-block-size configuration parameter exact meaning

Operations Configuration
1

Hi All,

Could anyone please explain/correct me what does write-block-size actually doing here My assumptions are

  1. As per documents it means block of size that will be written to persistence storage so suppose if i have a record having key → “random_str_1234” and value {“value”:1234567} resulting in lets say size of ~150 bytes so does it mean that record will be synced to disk only when it reaches to near around write-block-size?

  2. Or suppose i have defined block size of 128 KB does it mean first that 128Kb will be completely filled by records that i am writing and then aerospike will change the block to next available 128Kb space?

  3. Does this effect my storage usage any how? For eg. if i have this parameter configured to 128KB and aeropsike taking approx 45GB for 90 milion records so if i reduce this to lets say 64KB will aerospike use less storage space?

  4. Also if you can explain key size effect on storage usage that would be helpful as i tried changing length of my key to see effect on storgae usage but i didn’t see any effect on AMC ui on disk storage however when.i ran the backup job then i see the difference of 20 byte per record. Initially it was printing 173byte/rec and after changing key it printed 153byte/rec.

Thanks in advance!!

@kporter @Albot @meher @pgupta

2
  1. It’s when you’ve written enough records to fill 1 w-block OR if a record has been sitting in swb for more than flush-max-ms (default=1s) Configuration Reference | Aerospike Documentation . in your example if you are writing 150byte records, you’ll need to write about (128KiB / 150 byte) 873 records before the data is persisted (or 1s)
  2. Yes.
  3. It mainly affects the write-amplification of defragmentation, as well as performance. Not so much storage space.
  4. Key size? Key’s aren’t stored in Aerospike by default. There is an option to do it, though. By default the key is hashed and thats what gets stored, not your key. You can optionally store the key in writePolicy I think sendKey=true, or write it as its own bin is another common practice.
3

I would add that the write-block-size also impacts the post-write-queue as it is configured as a ‘number of write blocks’. This can have significant performance impact depending on your use case.

4

Thanks @meher and @Albot . yes @Albot we are using sendKey=true. so does the size is fixed for each record? for example suppose initially i was using below key value “RANDOM_MINI_CAB_30AUG_1234567_56879” → “123446677” and if i updates same to “1234567_56879” → “123446677” so it should take less storage space right as i have reduced the bytes at key part but i have not seen any change at amc side for same no. of records. if you please help me to crack that it would be helpful as we are looking to reduce aerospike storage usage.

5

This should be outlined in the capacity planning document here.

Assuming you are worried about persistent storage space:

If storing the record’s key. Flat key size is the exact opaque bytes sent by client:+ 1-3 bytes overhead (1 byte for key size <128 bytes, 2 bytes for <16K, 3 bytes for >= 16K) + 1 byte (key type overhead) + flat key size

1 Like
6

And, don’t forget there is a 16 bytes ‘rounding up’ for the storage… so if you do some smaller size reduction, you may not notice if you are still within the same ‘16 bytes window’.

1 Like