Thanks @pgupta
Its a burst work load, I don’t notice #1 but I notice #3. I already tried write-block-size to be 4xmyblocksize, it does fill 75% of the block. It is slightly better but not a significantly better.
Thanks for all the responses. Really appreciate it.
If you have burst load, you should also make sure you are not throttling on network bandwidth on your total network interface capacity - especially on AWS. Calculate total gigabits per second including a- incoming write, b - outgoing replica write c- incoming replica write from another node. Use your specific burst write rate and record size.