Confusing benchmark results

Benchmarks Aerospike Server Benchmarks
1

Hi, I’ve been trying to test Aerospike and have been getting peculiar results on AWS. I’m trying to run a benchmark on a c3.8xlarge instance and performance seems to degrade very quickly, I keep getting similar results on other instance types and using our own application as well as the benchmark tool. Is there some clear explanation for this that I’m missing? Should I be doing something different on AWS?

thanks.

here are my configuration and a sample of benchmark results(the storage device is at 3000 IOPS)

service {
        user root
        group root
        paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
        pidfile /var/run/aerospike/asd.pid
        service-threads 32
        transaction-queues 32
        transaction-threads-per-queue 3
        proto-fd-max 15000
}

namespace up {                              
        replication-factor 1                
        memory-size 8G                      
                                            
        storage-engine device {             
                device /dev/sdb             
        fsync-max-sec 120                   
                scheduler-mode noop         
                write-block-size 128K       
                                            
        }                                   
}           

./run_benchmarks -h 0.0.0.0 -p 3000 -n up -k 10000000 -S 1 -o S:50 -w RU,10 -z 20
2015-06-17 13:32:33.688 write(tps=83340 timeouts=0 errors=0) read(tps=9229 timeouts=0 errors=0) total(tps=92569 timeouts=0 errors=0)
2015-06-17 13:32:34.688 write(tps=80708 timeouts=0 errors=0) read(tps=9173 timeouts=0 errors=0) total(tps=89881 timeouts=0 errors=0)
2015-06-17 13:32:35.689 write(tps=80342 timeouts=0 errors=0) read(tps=8927 timeouts=0 errors=0) total(tps=89269 timeouts=0 errors=0)
2015-06-17 13:32:36.693 write(tps=77859 timeouts=0 errors=0) read(tps=8649 timeouts=0 errors=0) total(tps=86508 timeouts=0 errors=0)
2015-06-17 13:32:37.693 write(tps=72201 timeouts=0 errors=0) read(tps=8119 timeouts=0 errors=0) total(tps=80320 timeouts=0 errors=0)
2015-06-17 13:32:38.694 write(tps=49787 timeouts=0 errors=0) read(tps=5600 timeouts=0 errors=0) total(tps=55387 timeouts=0 errors=0)
2015-06-17 13:32:39.694 write(tps=46020 timeouts=0 errors=0) read(tps=5012 timeouts=0 errors=0) total(tps=51032 timeouts=0 errors=0)
2015-06-17 13:32:40.695 write(tps=44403 timeouts=0 errors=0) read(tps=4832 timeouts=0 errors=0) total(tps=49235 timeouts=0 errors=0)
2015-06-17 13:32:41.695 write(tps=40775 timeouts=0 errors=0) read(tps=4419 timeouts=0 errors=0) total(tps=45194 timeouts=0 errors=0)
2015-06-17 13:32:42.696 write(tps=39350 timeouts=0 errors=0) read(tps=4380 timeouts=0 errors=0) total(tps=43730 timeouts=0 errors=0)
2015-06-17 13:32:43.696 write(tps=36652 timeouts=0 errors=0) read(tps=4040 timeouts=0 errors=0) total(tps=40692 timeouts=0 errors=0)
2015-06-17 13:32:44.697 write(tps=34590 timeouts=0 errors=0) read(tps=3785 timeouts=0 errors=0) total(tps=38375 timeouts=0 errors=0)
2015-06-17 13:32:45.697 write(tps=35564 timeouts=0 errors=0) read(tps=3932 timeouts=0 errors=0) total(tps=39496 timeouts=0 errors=0)
2015-06-17 13:32:46.697 write(tps=33590 timeouts=0 errors=0) read(tps=3667 timeouts=0 errors=0) total(tps=37257 timeouts=0 errors=0)
2015-06-17 13:32:47.698 write(tps=31478 timeouts=0 errors=0) read(tps=3536 timeouts=0 errors=0) total(tps=35014 timeouts=0 errors=0)
2015-06-17 13:32:48.698 write(tps=29840 timeouts=0 errors=0) read(tps=3188 timeouts=0 errors=0) total(tps=33028 timeouts=0 errors=0)
2015-06-17 13:32:49.699 write(tps=27782 timeouts=0 errors=0) read(tps=3114 timeouts=0 errors=0) total(tps=30896 timeouts=0 errors=0)
2015-06-17 13:32:50.699 write(tps=28677 timeouts=0 errors=0) read(tps=3066 timeouts=0 errors=0) total(tps=31743 timeouts=0 errors=0)
2015-06-17 13:32:51.700 write(tps=28119 timeouts=0 errors=0) read(tps=3115 timeouts=0 errors=0) total(tps=31234 timeouts=0 errors=0)
2015-06-17 13:32:52.700 write(tps=29012 timeouts=0 errors=0) read(tps=3183 timeouts=0 errors=0) total(tps=32195 timeouts=0 errors=0)
2015-06-17 13:32:53.700 write(tps=27378 timeouts=0 errors=0) read(tps=2968 timeouts=0 errors=0) total(tps=30346 timeouts=0 errors=0)
2015-06-17 13:32:54.701 write(tps=25923 timeouts=0 errors=0) read(tps=2979 timeouts=0 errors=0) total(tps=28902 timeouts=0 errors=0)
2015-06-17 13:32:55.701 write(tps=25543 timeouts=0 errors=0) read(tps=2894 timeouts=0 errors=0) total(tps=28437 timeouts=0 errors=0)
2015-06-17 13:32:56.702 write(tps=25325 timeouts=0 errors=0) read(tps=2702 timeouts=0 errors=0) total(tps=28027 timeouts=0 errors=0)
2015-06-17 13:32:57.702 write(tps=25336 timeouts=0 errors=0) read(tps=2746 timeouts=0 errors=0) total(tps=28082 timeouts=0 errors=0)
2015-06-17 13:32:58.702 write(tps=23200 timeouts=0 errors=0) read(tps=2585 timeouts=0 errors=0) total(tps=25785 timeouts=0 errors=0)
2015-06-17 13:32:59.703 write(tps=23373 timeouts=0 errors=0) read(tps=2675 timeouts=0 errors=0) total(tps=26048 timeouts=0 errors=0)
2015-06-17 13:33:00.703 write(tps=22998 timeouts=0 errors=0) read(tps=2589 timeouts=0 errors=0) total(tps=25587 timeouts=0 errors=0)
2015-06-17 13:33:01.704 write(tps=23448 timeouts=0 errors=0) read(tps=2613 timeouts=0 errors=0) total(tps=26061 timeouts=0 errors=0)
2015-06-17 13:33:02.704 write(tps=22353 timeouts=0 errors=0) read(tps=2521 timeouts=0 errors=0) total(tps=24874 timeouts=0 errors=0)
2015-06-17 13:33:03.705 write(tps=22359 timeouts=0 errors=0) read(tps=2388 timeouts=0 errors=0) total(tps=24747 timeouts=0 errors=0)
2015-06-17 13:33:04.705 write(tps=22093 timeouts=0 errors=0) read(tps=2382 timeouts=0 errors=0) total(tps=24475 timeouts=0 errors=0)
2015-06-17 13:33:05.705 write(tps=22736 timeouts=0 errors=0) read(tps=2479 timeouts=0 errors=0) total(tps=25215 timeouts=0 errors=0)
2015-06-17 13:33:06.706 write(tps=20876 timeouts=0 errors=0) read(tps=2371 timeouts=0 errors=0) total(tps=23247 timeouts=0 errors=0)
2015-06-17 13:33:07.707 write(tps=20731 timeouts=0 errors=0) read(tps=2343 timeouts=0 errors=0) total(tps=23074 timeouts=0 errors=0)
2015-06-17 13:33:08.707 write(tps=20434 timeouts=0 errors=0) read(tps=2255 timeouts=0 errors=0) total(tps=22689 timeouts=0 errors=0)
2015-06-17 13:33:09.707 write(tps=20406 timeouts=0 errors=0) read(tps=2247 timeouts=0 errors=0) total(tps=22653 timeouts=0 errors=0)
2015-06-17 13:33:10.708 write(tps=20318 timeouts=0 errors=0) read(tps=2156 timeouts=0 errors=0) total(tps=22474 timeouts=0 errors=0)
2015-06-17 13:33:11.708 write(tps=20782 timeouts=0 errors=0) read(tps=2297 timeouts=0 errors=0) total(tps=23079 timeouts=0 errors=0)
2015-06-17 13:33:12.708 write(tps=19874 timeouts=0 errors=0) read(tps=2234 timeouts=0 errors=0) total(tps=22108 timeouts=0 errors=0)
2015-06-17 13:33:13.709 write(tps=19780 timeouts=0 errors=0) read(tps=2158 timeouts=0 errors=0) total(tps=21938 timeouts=0 errors=0)
2015-06-17 13:33:14.709 write(tps=19410 timeouts=0 errors=0) read(tps=2099 timeouts=0 errors=0) total(tps=21509 timeouts=0 errors=0)
2015-06-17 13:33:15.709 write(tps=18904 timeouts=0 errors=0) read(tps=2043 timeouts=0 errors=0) total(tps=20947 timeouts=0 errors=0)
2015-06-17 13:33:16.710 write(tps=19020 timeouts=0 errors=0) read(tps=2075 timeouts=0 errors=0) total(tps=21095 timeouts=0 errors=0)
2015-06-17 13:33:17.710 write(tps=18519 timeouts=0 errors=0) read(tps=2018 timeouts=0 errors=0) total(tps=20537 timeouts=0 errors=0)
2015-06-17 13:33:18.710 write(tps=18753 timeouts=0 errors=0) read(tps=1996 timeouts=0 errors=0) total(tps=20749 timeouts=0 errors=0)
2015-06-17 13:33:19.710 write(tps=18490 timeouts=0 errors=0) read(tps=1993 timeouts=0 errors=0) total(tps=20483 timeouts=0 errors=0)
2015-06-17 13:33:20.711 write(tps=18391 timeouts=0 errors=0) read(tps=2061 timeouts=0 errors=0) total(tps=20452 timeouts=0 errors=0)
2015-06-17 13:33:21.711 write(tps=17785 timeouts=0 errors=0) read(tps=2056 timeouts=0 errors=0) total(tps=19841 timeouts=0 errors=0)
2015-06-17 13:33:22.711 write(tps=19063 timeouts=0 errors=0) read(tps=2099 timeouts=0 errors=0) total(tps=21162 timeouts=0 errors=0)
2015-06-17 13:33:23.712 write(tps=18489 timeouts=0 errors=0) read(tps=2053 timeouts=0 errors=0) total(tps=20542 timeouts=0 errors=0)
2015-06-17 13:33:24.712 write(tps=17186 timeouts=0 errors=0) read(tps=1944 timeouts=0 errors=0) total(tps=19130 timeouts=0 errors=0)
2015-06-17 13:33:25.713 write(tps=17432 timeouts=0 errors=0) read(tps=1985 timeouts=0 errors=0) total(tps=19417 timeouts=0 errors=0)
2015-06-17 13:33:26.713 write(tps=16482 timeouts=0 errors=0) read(tps=1826 timeouts=0 errors=0) total(tps=18308 timeouts=0 errors=0)
2015-06-17 13:33:27.713 write(tps=16114 timeouts=0 errors=0) read(tps=1754 timeouts=0 errors=0) total(tps=17868 timeouts=0 errors=0)
2015-06-17 13:33:28.713 write(tps=17082 timeouts=0 errors=0) read(tps=1946 timeouts=0 errors=0) total(tps=19028 timeouts=0 errors=0)
3

So, the problem here is that the performance measured in tps degrades over time?

Does it stabilize at some point? I can image Amazon having some leaky-bucket limiter or some boosting algorithm on the promised IOPS…

4

I am assuming that you are using the local SSDs. Let me know if its otherwise. The local SSDs will give a burst of 3000 IOPS, not a guaranteed one. It can range from 24-3000 IOPS. You might be aware of it already. On a related note, the IOPS limit of i2.* instances are much higher. the i2.* machines are meant for high-I/O. You may want to take a look at them.

In your case, the reads started at about 9k tps but quickly dropped down to 2k tps and more or less stabilized there. Similarly, write tps started at about 90k tps but settled at about 20ktps. First, when using pure SSD storage, please note that each read is from the disk but writes are buffered and so all writes do not hit the disk (in simple words, multiple writes are collected together and flushed together).

It seems that AWS allowed you to go beyond the 3000IOPS, but quickly throttled to its allowed limit and hence the drop in the reads to about 2000 tps. My guess is that the other 1000 IOPS is consumed by the writes. If you can share the output of iostat, we will know the total IOPS done on the disk and it will give pretty good picture. If this is true, the tps that you are seeing here are more or less in line with the expectations.

Other way to check the above theory is to try with i2 instances which has much higher IOPS limit for the instance store.

1 Like