Hi, We are doing heavy continuous writes on AS Cluster with data on Azure NAS (for durability) for > 10 minutes.
With dd /dev/zero and SAR we see around 250MB/s of write throughput on our Azure NAS(Standard L8s)-1.5TB NAS
But with 128KB write-block-size, we get 60MB/s of throughput on NAS (as seen by SAR command) just before hitting “Device Overload”.
When we increased write-block-size of 512KB, we got 120MB/s of throughput on NAS(just before hitting “Device Overload”).
For write-block-size of 1024KB, we got 160MB/s of throughput on NAS. But with more increase in application threads we started hitting device overload issue(even when throughput between AS Node and NAS was around 160MB/s)
I have few questions about it:
- Why increasing Aerospike write block size, increases write throughput(in bytes) on Azure NAS(just before hitting “Device Overload” issue) ?
- Why “Device Overload” issues happen even when SAR shows write operations(and NO read ops) with throughput(160MB/s) which is far lesser than 250MB/s throughput(as seen by DD command) supported by NAS. Doesn’t “Device Overload” issue comes when writer thread not able to keep up with NAS speed
- Apart from “write-block-size”, what are other configs that can be varied to increase write throughput before hitting Device Overload issue ?
- Whats config param to increase number of threads that flush data from write-queue to block device
Note: For 1.5TB NAS, Azure supports ~5000 IOPS so with write-block-size of 1MB it should be around 5GB/s but due to limitation of 250MB/s, I was expecting atleast 250MB/s throughput before hitting Device Overload