Some questions about aerospike performance test

Benchmarks Aerospike Server Benchmarks
1

Hi everyone：

I am currently conducting some tests on aerospike, including HA and read and write performance.

I used 3 test servers, the hardware is nvme SSD 900G, 4vCPU, 32G memory.

Question 1 During my test, I simulated a downtime experiment. I manually forcibly killed the asd process on any server to see if it affected the test and how long the recovery time was.

But the reality is not particularly ideal. The test client started reporting errors.

2021-04-01 15:23:01.510 write(tps=20779 timeouts=0 errors=0) read(tps=82490 timeouts=0 
errors=0) total(tps=103269 timeouts=0 errors=0)
2021-04-01 15:23:02.511 write(tps=20633 timeouts=0 errors=0) read(tps=82298 timeouts=0 
errors=0) total(tps=102931 timeouts=0 errors=0)
2021-04-01 15:23:03.511 write(tps=20665 timeouts=0 errors=0) read(tps=82686 timeouts=0 
errors=0) total(tps=103351 timeouts=0 errors=0)
2021-04-01 15:23:04.512 write(tps=20771 timeouts=0 errors=0) read(tps=82328 timeouts=0 
errors=0) total(tps=103099 timeouts=0 errors=0)
2021-04-01 15:23:05.512 write(tps=13280 timeouts=0 errors=22) read(tps=52939 timeouts=0 
errors=0) total(tps=66219 timeouts=0 errors=22)
2021-04-01 15:23:05.615 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.io.EOFException
2021-04-01 15:23:06.512 write(tps=25 timeouts=20 errors=18) read(tps=231 timeouts=0 errors=0)    
total(tps=256 timeouts=20 errors=18)
2021-04-01 15:23:06.616 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:07.512 write(tps=4 timeouts=20 errors=12) read(tps=132 timeouts=0 errors=0) 
total(tps=136 timeouts=20 errors=12)
2021-04-01 15:23:07.617 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:08.513 write(tps=23 timeouts=18 errors=22) read(tps=233 timeouts=0 errors=0) 
total(tps=256 timeouts=18 errors=22)
2021-04-01 15:23:08.618 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:09.513 write(tps=15 timeouts=11 errors=7) read(tps=155 timeouts=0 errors=0) 
total(tps=170 timeouts=11 errors=7)
2021-04-01 15:23:09.619 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:10.513 write(tps=21 timeouts=15 errors=16) read(tps=197 timeouts=0 errors=0) 
total(tps=218 timeouts=15 errors=16)
2021-04-01 15:23:10.620 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:11.513 write(tps=7 timeouts=16 errors=15) read(tps=146 timeouts=0 errors=0) 
total(tps=153 timeouts=16 errors=15)
2021-04-01 15:23:11.621 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:12.514 write(tps=20 timeouts=20 errors=20) read(tps=291 timeouts=0 errors=0) t 
otal(tps=311 timeouts=20 errors=20)
2021-04-01 15:23:12.623 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:13.514 write(tps=17 timeouts=20 errors=18) read(tps=240 timeouts=0 errors=0) 
total(tps=257 timeouts=20 errors=18)
2021-04-01 15:23:13.624 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:14.514 write(tps=30 timeouts=20 errors=38) read(tps=378 timeouts=0 errors=0) 
total(tps=408 timeouts=20 errors=38)
2021-04-01 15:23:14.625 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:15.514 write(tps=6 timeouts=20 errors=11) read(tps=151 timeouts=0 errors=0) 
total(tps=157 timeouts=20 errors=11)
2021-04-01 15:23:15.626 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:16.515 write(tps=28 timeouts=20 errors=22) read(tps=233 timeouts=0 errors=0) 
total(tps=261 timeouts=20 errors=22)
2021-04-01 15:23:16.626 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:17.515 write(tps=18 timeouts=20 errors=21) read(tps=245 timeouts=0 errors=0) 
total(tps=263 timeouts=20 errors=21)
2021-04-01 15:23:17.627 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:18.515 write(tps=13 timeouts=20 errors=22) read(tps=231 timeouts=0 errors=0) 
total(tps=244 timeouts=20 errors=22)
2021-04-01 15:23:18.628 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:19.515 write(tps=10 timeouts=20 errors=12) read(tps=143 timeouts=0 errors=0) 
total(tps=153 timeouts=20 errors=12)
2021-04-01 15:23:19.629 WARN Thread tend Node BB93624143E1600 10.114.1.212 3000 refresh 
failed: Error -8: java.net.ConnectException: Connection refused (Connection refused)
2021-04-01 15:23:19.631 INFO Thread tend Remove node BB93624143E1600 10.114.1.212 3000
2021-04-01 15:23:20.515 write(tps=4342 timeouts=20 errors=0) read(tps=17261 timeouts=0 
errors=0) total(tps=21603 timeouts=20 errors=0)
2021-04-01 15:23:21.516 write(tps=15933 timeouts=0 errors=0) read(tps=62718 timeouts=0 
errors=0) total(tps=78651 timeouts=0 errors=0)
2021-04-01 15:23:22.516 write(tps=15738 timeouts=0 errors=0) read(tps=62865 timeouts=0 
errors=0) total(tps=78603 timeouts=0 errors=0)

Is there any solution that can solve the problem of any node downtime that will not affect the test client?

Question 2 Why does the number of tps decrease with the increasing number of records during my long time reading and writing test?

The number of tps for a single node is about one million records and starts to decline.

    [root@qa-aerospike-act aerospike]# bin/asbenchmark -h 10.114.1.210 -p 3000 -n test -k 10000000 -S 1 -o B:4096 -w RU,80 -z 20
Benchmark: 10.114.1.210 3000, namespace: test, set: testset, threads: 20, workload: READ_UPDATE
read: 80% (all bins: 100%, single bin: 0%), write: 20% (all bins: 100%, single bin: 0%)
keys: 10000000, start key: 1, transactions: 0, bins: 1, random values: false, throughput: unlimited
read policy:
    socketTimeout: 30000, totalTimeout: 0, timeoutDelay: 0, maxRetries: 2, sleepBetweenRetries: 0
    readModeAP: ONE, readModeSC: SESSION, replica: SEQUENCE, reportNotFound: false
write policy:
    socketTimeout: 30000, totalTimeout: 0, timeoutDelay: 0, maxRetries: 0, sleepBetweenRetries: 0
    commitLevel: COMMIT_ALL
Sync: connPoolsPerNode: 1
bin[0]: byte[4096]
debug: false
2021-04-02 16:23:40.339 INFO Thread main Add node BB97D9E363E1600 10.114.1.210 3000
2021-04-02 16:23:40.346 INFO Thread main Add node BB96C522E3E1600 10.114.1.211 3000
2021-04-02 16:23:40.346 INFO Thread main Add node BB93624143E1600 10.114.1.212 3000
2021-04-02 16:23:41.259 write(tps=13352 timeouts=0 errors=0) read(tps=53725 timeouts=0 errors=0) total(tps=67077 timeouts=0 errors=0)
2021-04-02 16:23:42.259 write(tps=23851 timeouts=0 errors=0) read(tps=96186 timeouts=0 errors=0) total(tps=120037 timeouts=0 errors=0)
2021-04-02 16:23:43.260 write(tps=24317 timeouts=0 errors=0) read(tps=97953 timeouts=0 errors=0) total(tps=122270 timeouts=0 errors=0)
2021-04-02 16:23:44.260 write(tps=24500 timeouts=0 errors=0) read(tps=98297 timeouts=0 errors=0) total(tps=122797 timeouts=0 errors=0)
2021-04-02 16:23:45.261 write(tps=24622 timeouts=0 errors=0) read(tps=98392 timeouts=0 errors=0) total(tps=123014 timeouts=0 errors=0)
2021-04-02 16:23:46.261 write(tps=24373 timeouts=0 errors=0) read(tps=97888 timeouts=0 errors=0) total(tps=122261 timeouts=0 errors=0)
2021-04-02 16:23:47.261 write(tps=24637 timeouts=0 errors=0) read(tps=97620 timeouts=0 errors=0) total(tps=122257 timeouts=0 errors=0)
2021-04-02 16:23:48.262 write(tps=24247 timeouts=0 errors=0) read(tps=97870 timeouts=0 errors=0) total(tps=122117 timeouts=0 errors=0)
2021-04-02 16:23:49.263 write(tps=24727 timeouts=0 errors=0) read(tps=98839 timeouts=0 errors=0) total(tps=123566 timeouts=0 errors=0)
2021-04-02 16:23:50.263 write(tps=24571 timeouts=0 errors=0) read(tps=99818 timeouts=0 errors=0) total(tps=124389 timeouts=0 errors=0)
2021-04-02 16:23:51.264 write(tps=24857 timeouts=0 errors=0) read(tps=98563 timeouts=0 errors=0) total(tps=123420 timeouts=0 errors=0)
2021-04-02 16:23:52.264 write(tps=24896 timeouts=0 errors=0) read(tps=99379 timeouts=0 errors=0) total(tps=124275 timeouts=0 errors=0)
2021-04-02 16:23:53.264 write(tps=25083 timeouts=0 errors=0) read(tps=98252 timeouts=0 errors=0) total(tps=123335 timeouts=0 errors=0)
2021-04-02 16:23:54.265 write(tps=24744 timeouts=0 errors=0) read(tps=98832 timeouts=0 errors=0) total(tps=123576 timeouts=0 errors=0)
2021-04-02 16:23:55.265 write(tps=24869 timeouts=0 errors=0) read(tps=98341 timeouts=0 errors=0) total(tps=123210 timeouts=0 errors=0)
2021-04-02 16:23:56.266 write(tps=24613 timeouts=0 errors=0) read(tps=98605 timeouts=0 errors=0) total(tps=123218 timeouts=0 errors=0)
2021-04-02 16:23:57.266 write(tps=24742 timeouts=0 errors=0) read(tps=98603 timeouts=0 errors=0) total(tps=123345 timeouts=0 errors=0)
2021-04-02 16:23:58.266 write(tps=24726 timeouts=0 errors=0) read(tps=98259 timeouts=0 errors=0) total(tps=122985 timeouts=0 errors=0)
2021-04-02 16:23:59.267 write(tps=24428 timeouts=0 errors=0) read(tps=98803 timeouts=0 errors=0) total(tps=123231 timeouts=0 errors=0)
2021-04-02 16:24:00.267 write(tps=24410 timeouts=0 errors=0) read(tps=98407 timeouts=0 errors=0) total(tps=122817 timeouts=0 errors=0)
2021-04-02 16:24:01.267 write(tps=24544 timeouts=0 errors=0) read(tps=98435 timeouts=0 errors=0) total(tps=122979 timeouts=0 errors=0)
2021-04-02 16:24:02.268 write(tps=24559 timeouts=0 errors=0) read(tps=98095 timeouts=0 errors=0) total(tps=122654 timeouts=0 errors=0)
2021-04-02 16:24:03.268 write(tps=24425 timeouts=0 errors=0) read(tps=98109 timeouts=0 errors=0) total(tps=122534 timeouts=0 errors=0)
2021-04-02 16:24:04.268 write(tps=24375 timeouts=0 errors=0) read(tps=97935 timeouts=0 errors=0) total(tps=122310 timeouts=0 errors=0)
2021-04-02 16:24:05.268 write(tps=24252 timeouts=0 errors=0) read(tps=97708 timeouts=0 errors=0) total(tps=121960 timeouts=0 errors=0)
2021-04-02 16:24:06.269 write(tps=24287 timeouts=0 errors=0) read(tps=97798 timeouts=0 errors=0) total(tps=122085 timeouts=0 errors=0)
2021-04-02 16:24:07.269 write(tps=24183 timeouts=0 errors=0) read(tps=97364 timeouts=0 errors=0) total(tps=121547 timeouts=0 errors=0)
2021-04-02 16:24:08.269 write(tps=24333 timeouts=0 errors=0) read(tps=97394 timeouts=0 errors=0) total(tps=121727 timeouts=0 errors=0)
2021-04-02 16:24:09.270 write(tps=24289 timeouts=0 errors=0) read(tps=96813 timeouts=0 errors=0) total(tps=121102 timeouts=0 errors=0)
2021-04-02 16:24:10.270 write(tps=24203 timeouts=0 errors=0) read(tps=96812 timeouts=0 errors=0) total(tps=121015 timeouts=0 errors=0)
2021-04-02 16:24:11.271 write(tps=24295 timeouts=0 errors=0) read(tps=96786 timeouts=0 errors=0) total(tps=121081 timeouts=0 errors=0)
2021-04-02 16:24:12.271 write(tps=24064 timeouts=0 errors=0) read(tps=97268 timeouts=0 errors=0) total(tps=121332 timeouts=0 errors=0)
2021-04-02 16:24:13.272 write(tps=24383 timeouts=0 errors=0) read(tps=96033 timeouts=0 errors=0) total(tps=120416 timeouts=0 errors=0)
2021-04-02 16:24:14.272 write(tps=24052 timeouts=0 errors=0) read(tps=96855 timeouts=0 errors=0) total(tps=120907 timeouts=0 errors=0)
2021-04-02 16:24:15.273 write(tps=24494 timeouts=0 errors=0) read(tps=95350 timeouts=0 errors=0) total(tps=119844 timeouts=0 errors=0)
2021-04-02 16:24:16.273 write(tps=24086 timeouts=0 errors=0) read(tps=96055 timeouts=0 errors=0) total(tps=120141 timeouts=0 errors=0)
2021-04-02 16:24:17.274 write(tps=24043 timeouts=0 errors=0) read(tps=95946 timeouts=0 errors=0) total(tps=119989 timeouts=0 errors=0)
2021-04-02 16:24:18.274 write(tps=24012 timeouts=0 errors=0) read(tps=96257 timeouts=0 errors=0) total(tps=120269 timeouts=0 errors=0)
2021-04-02 16:24:19.274 write(tps=24008 timeouts=0 errors=0) read(tps=96087 timeouts=0 errors=0) total(tps=120095 timeouts=0 errors=0)
2021-04-02 16:24:20.275 write(tps=23963 timeouts=0 errors=0) read(tps=95915 timeouts=0 errors=0) total(tps=119878 timeouts=0 errors=0)
2021-04-02 16:24:21.275 write(tps=23955 timeouts=0 errors=0) read(tps=95782 timeouts=0 errors=0) total(tps=119737 timeouts=0 errors=0)
2021-04-02 16:24:22.275 write(tps=23837 timeouts=0 errors=0) read(tps=94820 timeouts=0 errors=0) total(tps=118657 timeouts=0 errors=0)
2021-04-02 16:24:23.276 write(tps=23925 timeouts=0 errors=0) read(tps=95731 timeouts=0 errors=0) total(tps=119656 timeouts=0 errors=0)
2021-04-02 16:24:24.276 write(tps=23811 timeouts=0 errors=0) read(tps=95514 timeouts=0 errors=0) total(tps=119325 timeouts=0 errors=0)
2021-04-02 16:24:25.276 write(tps=23765 timeouts=0 errors=0) read(tps=95312 timeouts=0 errors=0) total(tps=119077 timeouts=0 errors=0)
2021-04-02 16:24:26.276 write(tps=23703 timeouts=0 errors=0) read(tps=95780 timeouts=0 errors=0) total(tps=119483 timeouts=0 errors=0)
2021-04-02 16:24:27.277 write(tps=23540 timeouts=0 errors=0) read(tps=95271 timeouts=0 errors=0) total(tps=118811 timeouts=0 errors=0)
2021-04-02 16:24:28.277 write(tps=23784 timeouts=0 errors=0) read(tps=95235 timeouts=0 errors=0) total(tps=119019 timeouts=0 errors=0)
2021-04-02 16:24:29.277 write(tps=23863 timeouts=0 errors=0) read(tps=94217 timeouts=0 errors=0) total(tps=118080 timeouts=0 errors=0)
2021-04-02 16:24:30.278 write(tps=23687 timeouts=0 errors=0) read(tps=94598 timeouts=0 errors=0) total(tps=118285 timeouts=0 errors=0)
2021-04-02 16:24:31.278 write(tps=23756 timeouts=0 errors=0) read(tps=94854 timeouts=0 errors=0) total(tps=118610 timeouts=0 errors=0)
2021-04-02 16:24:32.278 write(tps=23594 timeouts=0 errors=0) read(tps=95018 timeouts=0 errors=0) total(tps=118612 timeouts=0 errors=0)
2021-04-02 16:24:33.279 write(tps=23742 timeouts=0 errors=0) read(tps=93262 timeouts=0 errors=0) total(tps=117004 timeouts=0 errors=0)
2021-04-02 16:24:34.279 write(tps=23600 timeouts=0 errors=0) read(tps=94171 timeouts=0 errors=0) total(tps=117771 timeouts=0 errors=0)
2021-04-02 16:24:35.279 write(tps=23742 timeouts=0 errors=0) read(tps=94084 timeouts=0 errors=0) total(tps=117826 timeouts=0 errors=0)
2021-04-02 16:24:36.279 write(tps=23587 timeouts=0 errors=0) read(tps=94347 timeouts=0 errors=0) total(tps=117934 timeouts=0 errors=0)
2021-04-02 16:24:37.280 write(tps=23384 timeouts=0 errors=0) read(tps=93578 timeouts=0 errors=0) total(tps=116962 timeouts=0 errors=0)
2021-04-02 16:24:38.280 write(tps=23534 timeouts=0 errors=0) read(tps=93314 timeouts=0 errors=0) total(tps=116848 timeouts=0 errors=0)
2021-04-02 16:24:39.280 write(tps=23480 timeouts=0 errors=0) read(tps=93187 timeouts=0 errors=0) total(tps=116667 timeouts=0 errors=0)
2021-04-02 16:24:40.281 write(tps=23476 timeouts=0 errors=0) read(tps=93385 timeouts=0 errors=0) total(tps=116861 timeouts=0 errors=0)
2021-04-02 16:24:41.281 write(tps=23270 timeouts=0 errors=0) read(tps=94072 timeouts=0 errors=0) total(tps=117342 timeouts=0 errors=0)
2021-04-02 16:24:42.281 write(tps=23370 timeouts=0 errors=0) read(tps=93766 timeouts=0 errors=0) total(tps=117136 timeouts=0 errors=0)
2021-04-02 16:24:43.281 write(tps=23193 timeouts=0 errors=0) read(tps=93709 timeouts=0 errors=0) total(tps=116902 timeouts=0 errors=0)
2021-04-02 16:24:44.282 write(tps=23292 timeouts=0 errors=0) read(tps=93610 timeouts=0 errors=0) total(tps=116902 timeouts=0 errors=0)
2021-04-02 16:24:45.282 write(tps=23285 timeouts=0 errors=0) read(tps=93056 timeouts=0 errors=0) total(tps=116341 timeouts=0 errors=0)
2021-04-02 16:24:46.282 write(tps=23331 timeouts=0 errors=0) read(tps=93050 timeouts=0 errors=0) total(tps=116381 timeouts=0 errors=0)
2021-04-02 16:24:47.282 write(tps=23109 timeouts=0 errors=0) read(tps=93579 timeouts=0 errors=0) total(tps=116688 timeouts=0 errors=0)
2021-04-02 16:24:48.283 write(tps=23372 timeouts=0 errors=0) read(tps=92889 timeouts=0 errors=0) total(tps=116261 timeouts=0 errors=0)
2021-04-02 16:24:49.283 write(tps=23266 timeouts=0 errors=0) read(tps=92187 timeouts=0 errors=0) total(tps=115453 timeouts=0 errors=0)
2021-04-02 16:24:50.283 write(tps=22855 timeouts=0 errors=0) read(tps=93535 timeouts=0 errors=0) total(tps=116390 timeouts=0 errors=0)
2021-04-02 16:24:51.283 write(tps=23277 timeouts=0 errors=0) read(tps=92706 timeouts=0 errors=0) total(tps=115983 timeouts=0 errors=0)
2021-04-02 16:24:52.284 write(tps=23051 timeouts=0 errors=0) read(tps=92252 timeouts=0 errors=0) total(tps=115303 timeouts=0 errors=0)
2021-04-02 16:24:53.284 write(tps=23046 timeouts=0 errors=0) read(tps=92530 timeouts=0 errors=0) total(tps=115576 timeouts=0 errors=0)
2021-04-02 16:24:54.284 write(tps=23100 timeouts=0 errors=0) read(tps=92441 timeouts=0 errors=0) total(tps=115541 timeouts=0 errors=0)
2021-04-02 16:24:55.285 write(tps=23075 timeouts=0 errors=0) read(tps=92241 timeouts=0 errors=0) total(tps=115316 timeouts=0 errors=0)
2021-04-02 16:24:56.285 write(tps=23153 timeouts=0 errors=0) read(tps=92595 timeouts=0 errors=0) total(tps=115748 timeouts=0 errors=0)
2021-04-02 16:24:57.285 write(tps=23110 timeouts=0 errors=0) read(tps=92309 timeouts=0 errors=0) total(tps=115419 timeouts=0 errors=0)
2021-04-02 16:24:58.285 write(tps=23192 timeouts=0 errors=0) read(tps=92361 timeouts=0 errors=0) total(tps=115553 timeouts=0 errors=0)
2021-04-02 16:24:59.286 write(tps=23116 timeouts=0 errors=0) read(tps=91525 timeouts=0 errors=0) total(tps=114641 timeouts=0 errors=0)
2021-04-02 16:25:00.286 write(tps=23131 timeouts=0 errors=0) read(tps=91713 timeouts=0 errors=0) total(tps=114844 timeouts=0 errors=0)
2021-04-02 16:25:01.286 write(tps=23229 timeouts=0 errors=0) read(tps=91665 timeouts=0 errors=0) total(tps=114894 timeouts=0 errors=0)
2021-04-02 16:25:02.287 write(tps=22956 timeouts=0 errors=0) read(tps=91605 timeouts=0 errors=0) total(tps=114561 timeouts=0 errors=0)
2021-04-02 16:25:03.287 write(tps=22765 timeouts=0 errors=0) read(tps=91783 timeouts=0 errors=0) total(tps=114548 timeouts=0 errors=0)
2021-04-02 16:25:04.287 write(tps=22918 timeouts=0 errors=0) read(tps=91957 timeouts=0 errors=0) total(tps=114875 timeouts=0 errors=0)
2021-04-02 16:25:05.287 write(tps=23009 timeouts=0 errors=0) read(tps=91491 timeouts=0 errors=0) total(tps=114500 timeouts=0 errors=0)
2021-04-02 16:25:06.288 write(tps=22951 timeouts=0 errors=0) read(tps=91362 timeouts=0 errors=0) total(tps=114313 timeouts=0 errors=0)
2021-04-02 16:25:07.288 write(tps=22761 timeouts=0 errors=0) read(tps=90917 timeouts=0 errors=0) total(tps=113678 timeouts=0 errors=0)
2021-04-02 16:25:08.288 write(tps=22900 timeouts=0 errors=0) read(tps=91438 timeouts=0 errors=0) total(tps=114338 timeouts=0 errors=0)
2021-04-02 16:25:09.291 write(tps=22982 timeouts=0 errors=0) read(tps=91223 timeouts=0 errors=0) total(tps=114205 timeouts=0 errors=0)
2021-04-02 16:25:10.292 write(tps=22697 timeouts=0 errors=0) read(tps=91298 timeouts=0 errors=0) total(tps=113995 timeouts=0 errors=0)
2021-04-02 16:25:11.292 write(tps=22621 timeouts=0 errors=0) read(tps=91935 timeouts=0 errors=0) total(tps=114556 timeouts=0 errors=0)
2021-04-02 16:25:12.292 write(tps=22842 timeouts=0 errors=0) read(tps=91142 timeouts=0 errors=0) total(tps=113984 timeouts=0 errors=0)
2021-04-02 16:25:13.292 write(tps=22970 timeouts=0 errors=0) read(tps=90902 timeouts=0 errors=0) total(tps=113872 timeouts=0 errors=0)
2021-04-02 16:25:14.292 write(tps=22772 timeouts=0 errors=0) read(tps=90772 timeouts=0 errors=0) total(tps=113544 timeouts=0 errors=0)
2021-04-02 16:25:15.293 write(tps=22667 timeouts=0 errors=0) read(tps=90890 timeouts=0 errors=0) total(tps=113557 timeouts=0 errors=0)
2021-04-02 16:25:16.293 write(tps=22463 timeouts=0 errors=0) read(tps=91253 timeouts=0 errors=0) total(tps=113716 timeouts=0 errors=0)
2021-04-02 16:25:17.293 write(tps=22507 timeouts=0 errors=0) read(tps=90124 timeouts=0 errors=0) total(tps=112631 timeouts=0 errors=0)
2021-04-02 16:25:18.294 write(tps=22713 timeouts=0 errors=0) read(tps=90683 timeouts=0 errors=0) total(tps=113396 timeouts=0 errors=0)
2021-04-02 16:25:19.294 write(tps=22277 timeouts=0 errors=0) read(tps=90665 timeouts=0 errors=0) total(tps=112942 timeouts=0 errors=0)
2021-04-02 16:25:20.294 write(tps=22586 timeouts=0 errors=0) read(tps=89837 timeouts=0 errors=0) total(tps=112423 timeouts=0 errors=0)
2021-04-02 16:25:21.294 write(tps=22341 timeouts=0 errors=0) read(tps=90795 timeouts=0 errors=0) total(tps=113136 timeouts=0 errors=0)
2021-04-02 16:25:22.294 write(tps=22585 timeouts=0 errors=0) read(tps=90033 timeouts=0 errors=0) total(tps=112618 timeouts=0 errors=0)
2021-04-02 16:25:23.295 write(tps=22580 timeouts=0 errors=0) read(tps=90302 timeouts=0 errors=0) total(tps=112882 timeouts=0 errors=0)
2021-04-02 16:25:24.295 write(tps=22667 timeouts=0 errors=0) read(tps=89871 timeouts=0 errors=0) total(tps=112538 timeouts=0 errors=0)
2021-04-02 16:25:25.295 write(tps=22372 timeouts=0 errors=0) read(tps=90632 timeouts=0 errors=0) total(tps=113004 timeouts=0 errors=0)
2021-04-02 16:25:26.295 write(tps=22615 timeouts=0 errors=0) read(tps=89864 timeouts=0 errors=0) total(tps=112479 timeouts=0 errors=0)
2021-04-02 16:25:27.296 write(tps=22541 timeouts=0 errors=0) read(tps=90154 timeouts=0 errors=0) total(tps=112695 timeouts=0 errors=0)
2021-04-02 16:25:28.296 write(tps=22476 timeouts=0 errors=0) read(tps=90183 timeouts=0 errors=0) total(tps=112659 timeouts=0 errors=0)
2021-04-02 16:25:29.296 write(tps=22398 timeouts=0 errors=0) read(tps=89701 timeouts=0 errors=0) total(tps=112099 timeouts=0 errors=0)
2021-04-02 16:25:30.296 write(tps=22465 timeouts=0 errors=0) read(tps=89357 timeouts=0 errors=0) total(tps=111822 timeouts=0 errors=0)
2021-04-02 16:25:31.297 write(tps=22225 timeouts=0 errors=0) read(tps=89553 timeouts=0 errors=0) total(tps=111778 timeouts=0 errors=0)
2021-04-02 16:25:32.297 write(tps=22359 timeouts=0 errors=0) read(tps=89171 timeouts=0 errors=0) total(tps=111530 timeouts=0 errors=0)
2021-04-02 16:25:33.297 write(tps=22412 timeouts=0 errors=0) read(tps=89790 timeouts=0 errors=0) total(tps=112202 timeouts=0 errors=0)
2021-04-02 16:25:34.297 write(tps=22531 timeouts=0 errors=0) read(tps=89017 timeouts=0 errors=0) total(tps=111548 timeouts=0 errors=0)
2021-04-02 16:25:35.298 write(tps=22316 timeouts=0 errors=0) read(tps=89513 timeouts=0 errors=0) total(tps=111829 timeouts=0 errors=0)
2021-04-02 16:25:36.298 write(tps=22280 timeouts=0 errors=0) read(tps=89398 timeouts=0 errors=0) total(tps=111678 timeouts=0 errors=0)
2021-04-02 16:25:37.298 write(tps=22330 timeouts=0 errors=0) read(tps=88727 timeouts=0 errors=0) total(tps=111057 timeouts=0 errors=0)
2021-04-02 16:25:38.298 write(tps=22171 timeouts=0 errors=0) read(tps=89473 timeouts=0 errors=0) total(tps=111644 timeouts=0 errors=0)
2021-04-02 16:25:39.299 write(tps=22249 timeouts=0 errors=0) read(tps=88672 timeouts=0 errors=0) total(tps=110921 timeouts=0 errors=0)
2021-04-02 16:25:40.299 write(tps=22202 timeouts=0 errors=0) read(tps=88948 timeouts=0 errors=0) total(tps=111150 timeouts=0 errors=0)
2021-04-02 16:25:41.299 write(tps=22068 timeouts=0 errors=0) read(tps=89347 timeouts=0 errors=0) total(tps=111415 timeouts=0 errors=0)
2021-04-02 16:25:42.300 write(tps=22055 timeouts=0 errors=0) read(tps=88898 timeouts=0 errors=0) total(tps=110953 timeouts=0 errors=0)
2021-04-02 16:25:43.300 write(tps=22309 timeouts=0 errors=0) read(tps=88413 timeouts=0 errors=0) total(tps=110722 timeouts=0 errors=0)
2021-04-02 16:25:44.300 write(tps=22116 timeouts=0 errors=0) read(tps=89445 timeouts=0 errors=0) total(tps=111561 timeouts=0 errors=0)
2021-04-02 16:25:45.301 write(tps=22265 timeouts=0 errors=0) read(tps=88719 timeouts=0 errors=0) total(tps=110984 timeouts=0 errors=0)
2021-04-02 16:25:46.301 write(tps=22257 timeouts=0 errors=0) read(tps=88660 timeouts=0 errors=0) total(tps=110917 timeouts=0 errors=0)
2021-04-02 16:25:47.301 write(tps=22304 timeouts=0 errors=0) read(tps=88312 timeouts=0 errors=0) total(tps=110616 timeouts=0 errors=0)
2021-04-02 16:25:48.305 write(tps=22127 timeouts=0 errors=0) read(tps=88689 timeouts=0 errors=0) total(tps=110816 timeouts=0 errors=0)
2021-04-02 16:25:49.305 write(tps=22118 timeouts=0 errors=0) read(tps=88270 timeouts=0 errors=0) total(tps=110388 timeouts=0 errors=0)
2021-04-02 16:25:50.306 write(tps=22108 timeouts=0 errors=0) read(tps=88663 timeouts=0 errors=0) total(tps=110771 timeouts=0 errors=0)
2021-04-02 16:25:51.306 write(tps=22269 timeouts=0 errors=0) read(tps=88042 timeouts=0 errors=0) total(tps=110311 timeouts=0 errors=0)
2021-04-02 16:25:52.306 write(tps=22045 timeouts=0 errors=0) read(tps=87732 timeouts=0 errors=0) total(tps=109777 timeouts=0 errors=0)
2021-04-02 16:25:53.307 write(tps=21902 timeouts=0 errors=0) read(tps=88277 timeouts=0 errors=0) total(tps=110179 timeouts=0 errors=0)
2021-04-02 16:25:54.307 write(tps=21951 timeouts=0 errors=0) read(tps=88128 timeouts=0 errors=0) total(tps=110079 timeouts=0 errors=0)
2021-04-02 16:25:55.307 write(tps=22047 timeouts=0 errors=0) read(tps=88122 timeouts=0 errors=0) total(tps=110169 timeouts=0 errors=0)
2021-04-02 16:25:56.307 write(tps=22092 timeouts=0 errors=0) read(tps=88091 timeouts=0 errors=0) total(tps=110183 timeouts=0 errors=0)
2021-04-02 16:25:57.308 write(tps=22150 timeouts=0 errors=0) read(tps=88083 timeouts=0 errors=0) total(tps=110233 timeouts=0 errors=0)
2021-04-02 16:25:58.308 write(tps=21780 timeouts=0 errors=0) read(tps=87305 timeouts=0 errors=0) total(tps=109085 timeouts=0 errors=0)
2021-04-02 16:25:59.308 write(tps=21896 timeouts=0 errors=0) read(tps=87775 timeouts=0 errors=0) total(tps=109671 timeouts=0 errors=0)
2021-04-02 16:26:00.308 write(tps=21892 timeouts=0 errors=0) read(tps=87954 timeouts=0 errors=0) total(tps=109846 timeouts=0 errors=0)
2021-04-02 16:26:01.309 write(tps=21936 timeouts=0 errors=0) read(tps=87711 timeouts=0 errors=0) total(tps=109647 timeouts=0 errors=0)
2021-04-02 16:26:02.309 write(tps=22044 timeouts=0 errors=0) read(tps=87414 timeouts=0 errors=0) total(tps=109458 timeouts=0 errors=0)
2021-04-02 16:26:03.309 write(tps=21864 timeouts=0 errors=0) read(tps=87825 timeouts=0 errors=0) total(tps=109689 timeouts=0 errors=0)
2021-04-02 16:26:04.309 write(tps=21857 timeouts=0 errors=0) read(tps=87748 timeouts=0 errors=0) total(tps=109605 timeouts=0 errors=0)
2021-04-02 16:26:05.310 write(tps=21985 timeouts=0 errors=0) read(tps=87367 timeouts=0 errors=0) total(tps=109352 timeouts=0 errors=0)
2021-04-02 16:26:06.310 write(tps=21948 timeouts=0 errors=0) read(tps=87264 timeouts=0 errors=0) total(tps=109212 timeouts=0 errors=0)
2021-04-02 16:26:07.310 write(tps=21652 timeouts=0 errors=0) read(tps=87368 timeouts=0 errors=0) total(tps=109020 timeouts=0 errors=0)
2021-04-02 16:26:08.310 write(tps=21701 timeouts=0 errors=0) read(tps=87446 timeouts=0 errors=0) total(tps=109147 timeouts=0 errors=0)
2021-04-02 16:26:09.310 write(tps=21760 timeouts=0 errors=0) read(tps=86771 timeouts=0 errors=0) total(tps=108531 timeouts=0 errors=0)
2021-04-02 16:26:10.311 write(tps=21795 timeouts=0 errors=0) read(tps=86864 timeouts=0 errors=0) total(tps=108659 timeouts=0 errors=0)
2021-04-02 16:26:11.311 write(tps=21706 timeouts=0 errors=0) read(tps=87739 timeouts=0 errors=0) total(tps=109445 timeouts=0 errors=0)
2021-04-02 16:26:12.311 write(tps=21692 timeouts=0 errors=0) read(tps=86894 timeouts=0 errors=0) total(tps=108586 timeouts=0 errors=0)
2021-04-02 16:26:13.311 write(tps=21973 timeouts=0 errors=0) read(tps=86313 timeouts=0 errors=0) total(tps=108286 timeouts=0 errors=0)
2021-04-02 16:26:14.312 write(tps=21650 timeouts=0 errors=0) read(tps=86848 timeouts=0 errors=0) total(tps=108498 timeouts=0 errors=0)
2021-04-02 16:26:15.312 write(tps=21594 timeouts=0 errors=0) read(tps=86369 timeouts=0 errors=0) total(tps=107963 timeouts=0 errors=0)
2021-04-02 16:26:16.312 write(tps=21663 timeouts=0 errors=0) read(tps=86627 timeouts=0 errors=0) total(tps=108290 timeouts=0 errors=0)
2021-04-02 16:26:17.312 write(tps=21856 timeouts=0 errors=0) read(tps=86458 timeouts=0 errors=0) total(tps=108314 timeouts=0 errors=0)
2021-04-02 16:26:18.313 write(tps=21978 timeouts=0 errors=0) read(tps=86511 timeouts=0 errors=0) total(tps=108489 timeouts=0 errors=0)
2021-04-02 16:26:19.313 write(tps=21683 timeouts=0 errors=0) read(tps=86491 timeouts=0 errors=0) total(tps=108174 timeouts=0 errors=0)
2021-04-02 16:26:20.313 write(tps=21638 timeouts=0 errors=0) read(tps=86385 timeouts=0 errors=0) total(tps=108023 timeouts=0 errors=0)
2021-04-02 16:26:21.313 write(tps=21550 timeouts=0 errors=0) read(tps=86753 timeouts=0 errors=0) total(tps=108303 timeouts=0 errors=0)
2021-04-02 16:26:22.313 write(tps=21393 timeouts=0 errors=0) read(tps=86077 timeouts=0 errors=0) total(tps=107470 timeouts=0 errors=0)
2021-04-02 16:26:23.314 write(tps=21538 timeouts=0 errors=0) read(tps=86226 timeouts=0 errors=0) total(tps=107764 timeouts=0 errors=0)
2021-04-02 16:26:24.314 write(tps=21563 timeouts=0 errors=0) read(tps=86170 timeouts=0 errors=0) total(tps=107733 timeouts=0 errors=0)
2021-04-02 16:26:25.314 write(tps=21515 timeouts=0 errors=0) read(tps=86313 timeouts=0 errors=0) total(tps=107828 timeouts=0 errors=0)
2021-04-02 16:26:26.314 write(tps=21436 timeouts=0 errors=0) read(tps=86263 timeouts=0 errors=0) total(tps=107699 timeouts=0 errors=0)
2021-04-02 16:26:27.315 write(tps=21667 timeouts=0 errors=0) read(tps=85897 timeouts=0 errors=0) total(tps=107564 timeouts=0 errors=0)
2021-04-02 16:26:28.315 write(tps=21258 timeouts=0 errors=0) read(tps=85615 timeouts=0 errors=0) total(tps=106873 timeouts=0 errors=0)
2021-04-02 16:26:29.315 write(tps=21366 timeouts=0 errors=0) read(tps=85109 timeouts=0 errors=0) total(tps=106475 timeouts=0 errors=0)
2021-04-02 16:26:30.315 write(tps=21542 timeouts=0 errors=0) read(tps=85668 timeouts=0 errors=0) total(tps=107210 timeouts=0 errors=0)
2021-04-02 16:26:31.315 write(tps=21642 timeouts=0 errors=0) read(tps=85505 timeouts=0 errors=0) total(tps=107147 timeouts=0 errors=0)
2021-04-02 16:26:32.316 write(tps=21319 timeouts=0 errors=0) read(tps=85732 timeouts=0 errors=0) total(tps=107051 timeouts=0 errors=0)
2021-04-02 16:26:33.316 write(tps=21527 timeouts=0 errors=0) read(tps=85609 timeouts=0 errors=0) total(tps=107136 timeouts=0 errors=0)
2021-04-02 16:26:34.316 write(tps=21403 timeouts=0 errors=0) read(tps=85906 timeouts=0 errors=0) total(tps=107309 timeouts=0 errors=0)
2021-04-02 16:26:35.316 write(tps=21317 timeouts=0 errors=0) read(tps=86007 timeouts=0 errors=0) total(tps=107324 timeouts=0 errors=0)
2021-04-02 16:26:36.317 write(tps=21379 timeouts=0 errors=0) read(tps=85640 timeouts=0 errors=0) total(tps=107019 timeouts=0 errors=0)
2021-04-02 16:26:37.317 write(tps=21215 timeouts=0 errors=0) read(tps=85689 timeouts=0 errors=0) total(tps=106904 timeouts=0 errors=0)
2021-04-02 16:26:38.317 write(tps=21699 timeouts=0 errors=0) read(tps=85078 timeouts=0 errors=0) total(tps=106777 timeouts=0 errors=0)
2021-04-02 16:26:39.317 write(tps=21406 timeouts=0 errors=0) read(tps=85559 timeouts=0 errors=0) total(tps=106965 timeouts=0 errors=0)
2021-04-02 16:26:40.317 write(tps=21432 timeouts=0 errors=0) read(tps=85374 timeouts=0 errors=0) total(tps=106806 timeouts=0 errors=0)
2021-04-02 16:26:41.318 write(tps=21445 timeouts=0 errors=0) read(tps=85146 timeouts=0 errors=0) total(tps=106591 timeouts=0 errors=0)
2021-04-02 16:26:42.318 write(tps=21242 timeouts=0 errors=0) read(tps=85528 timeouts=0 errors=0) total(tps=106770 timeouts=0 errors=0)
2021-04-02 16:26:43.318 write(tps=21591 timeouts=0 errors=0) read(tps=84699 timeouts=0 errors=0) total(tps=106290 timeouts=0 errors=0)
2021-04-02 16:26:44.318 write(tps=21211 timeouts=0 errors=0) read(tps=85427 timeouts=0 errors=0) total(tps=106638 timeouts=0 errors=0)
2021-04-02 16:26:45.319 write(tps=21279 timeouts=0 errors=0) read(tps=85247 timeouts=0 errors=0) total(tps=106526 timeouts=0 errors=0)
2021-04-02 16:26:46.319 write(tps=21409 timeouts=0 errors=0) read(tps=84594 timeouts=0 errors=0) total(tps=106003 timeouts=0 errors=0)
2021-04-02 16:26:47.319 write(tps=21258 timeouts=0 errors=0) read(tps=85137 timeouts=0 errors=0) total(tps=106395 timeouts=0 errors=0)
2021-04-02 16:26:48.319 write(tps=21481 timeouts=0 errors=0) read(tps=84652 timeouts=0 errors=0) total(tps=106133 timeouts=0 errors=0)
2021-04-02 16:26:49.319 write(tps=21211 timeouts=0 errors=0) read(tps=84920 timeouts=0 errors=0) total(tps=106131 timeouts=0 errors=0)
2021-04-02 16:26:50.320 write(tps=21201 timeouts=0 errors=0) read(tps=85115 timeouts=0 errors=0) total(tps=106316 timeouts=0 errors=0)
2021-04-02 16:26:51.320 write(tps=21325 timeouts=0 errors=0) read(tps=85060 timeouts=0 errors=0) total(tps=106385 timeouts=0 errors=0)
2021-04-02 16:26:52.320 write(tps=21117 timeouts=0 errors=0) read(tps=84784 timeouts=0 errors=0) total(tps=105901 timeouts=0 errors=0)
2021-04-02 16:26:53.320 write(tps=21296 timeouts=0 errors=0) read(tps=84828 timeouts=0 errors=0) total(tps=106124 timeouts=0 errors=0)
2021-04-02 16:26:54.321 write(tps=21340 timeouts=0 errors=0) read(tps=84417 timeouts=0 errors=0) total(tps=105757 timeouts=0 errors=0)
2021-04-02 16:26:55.321 write(tps=21297 timeouts=0 errors=0) read(tps=84827 timeouts=0 errors=0) total(tps=106124 timeouts=0 errors=0)
2021-04-02 16:26:56.321 write(tps=21127 timeouts=0 errors=0) read(tps=84710 timeouts=0 errors=0) total(tps=105837 timeouts=0 errors=0)
2021-04-02 16:26:57.321 write(tps=20965 timeouts=0 errors=0) read(tps=84886 timeouts=0 errors=0) total(tps=105851 timeouts=0 errors=0)
2021-04-02 16:26:58.321 write(tps=20916 timeouts=0 errors=0) read(tps=84638 timeouts=0 errors=0) total(tps=105554 timeouts=0 errors=0)
2021-04-02 16:26:59.322 write(tps=20982 timeouts=0 errors=0) read(tps=84300 timeouts=0 errors=0) total(tps=105282 timeouts=0 errors=0)
2021-04-02 16:27:00.322 write(tps=20816 timeouts=0 errors=0) read(tps=83066 timeouts=0 errors=0) total(tps=103882 timeouts=0 errors=0)
2021-04-02 16:27:01.322 write(tps=20943 timeouts=0 errors=0) read(tps=84665 timeouts=0 errors=0) total(tps=105608 timeouts=0 errors=0)
2021-04-02 16:27:02.322 write(tps=20921 timeouts=0 errors=0) read(tps=84494 timeouts=0 errors=0) total(tps=105415 timeouts=0 errors=0)
2021-04-02 16:27:03.323 write(tps=20790 timeouts=0 errors=0) read(tps=83963 timeouts=0 errors=0) total(tps=104753 timeouts=0 errors=0)
2021-04-02 16:27:04.323 write(tps=20921 timeouts=0 errors=0) read(tps=83758 timeouts=0 errors=0) total(tps=104679 timeouts=0 errors=0)
2021-04-02 16:27:05.323 write(tps=20997 timeouts=0 errors=0) read(tps=84062 timeouts=0 errors=0) total(tps=105059 timeouts=0 errors=0)
2021-04-02 16:27:06.323 write(tps=21280 timeouts=0 errors=0) read(tps=83787 timeouts=0 errors=0) total(tps=105067 timeouts=0 errors=0)
2021-04-02 16:27:07.324 write(tps=20800 timeouts=0 errors=0) read(tps=84248 timeouts=0 errors=0) total(tps=105048 timeouts=0 errors=0)
2021-04-02 16:27:08.324 write(tps=21040 timeouts=0 errors=0) read(tps=84042 timeouts=0 errors=0) total(tps=105082 timeouts=0 errors=0)
2021-04-02 16:27:09.324 write(tps=20874 timeouts=0 errors=0) read(tps=83458 timeouts=0 errors=0) total(tps=104332 timeouts=0 errors=0)
2021-04-02 16:27:10.324 write(tps=21021 timeouts=0 errors=0) read(tps=83965 timeouts=0 errors=0) total(tps=104986 timeouts=0 errors=0)
2021-04-02 16:27:11.324 write(tps=21036 timeouts=0 errors=0) read(tps=83867 timeouts=0 errors=0) total(tps=104903 timeouts=0 errors=0)
2021-04-02 16:27:12.325 write(tps=20981 timeouts=0 errors=0) read(tps=83951 timeouts=0 errors=0) total(tps=104932 timeouts=0 errors=0)
2021-04-02 16:27:13.325 write(tps=21236 timeouts=0 errors=0) read(tps=83340 timeouts=0 errors=0) total(tps=104576 timeouts=0 errors=0)
2021-04-02 16:27:14.325 write(tps=20920 timeouts=0 errors=0) read(tps=83544 timeouts=0 errors=0) total(tps=104464 timeouts=0 errors=0)
2021-04-02 16:27:15.325 write(tps=20970 timeouts=0 errors=0) read(tps=83465 timeouts=0 errors=0) total(tps=104435 timeouts=0 errors=0)
2021-04-02 16:27:16.326 write(tps=20851 timeouts=0 errors=0) read(tps=83289 timeouts=0 errors=0) total(tps=104140 timeouts=0 errors=0)
2021-04-02 16:27:17.326 write(tps=20846 timeouts=0 errors=0) read(tps=83949 timeouts=0 errors=0) total(tps=104795 timeouts=0 errors=0)
2021-04-02 16:27:18.326 write(tps=21052 timeouts=0 errors=0) read(tps=83609 timeouts=0 errors=0) total(tps=104661 timeouts=0 errors=0)
2021-04-02 16:27:19.326 write(tps=20752 timeouts=0 errors=0) read(tps=83357 timeouts=0 errors=0) total(tps=104109 timeouts=0 errors=0)
2021-04-02 16:27:20.326 write(tps=20762 timeouts=0 errors=0) read(tps=83355 timeouts=0 errors=0) total(tps=104117 timeouts=0 errors=0)
2021-04-02 16:27:21.327 write(tps=20682 timeouts=0 errors=0) read(tps=83642 timeouts=0 errors=0) total(tps=104324 timeouts=0 errors=0)
2021-04-02 16:27:22.327 write(tps=20847 timeouts=0 errors=0) read(tps=83251 timeouts=0 errors=0) total(tps=104098 timeouts=0 errors=0)
2021-04-02 16:27:23.327 write(tps=20700 timeouts=0 errors=0) read(tps=83577 timeouts=0 errors=0) total(tps=104277 timeouts=0 errors=0)
2021-04-02 16:27:24.327 write(tps=20914 timeouts=0 errors=0) read(tps=83122 timeouts=0 errors=0) total(tps=104036 timeouts=0 errors=0)
2021-04-02 16:27:25.328 write(tps=20819 timeouts=0 errors=0) read(tps=83036 timeouts=0 errors=0) total(tps=103855 timeouts=0 errors=0)
2021-04-02 16:27:26.328 write(tps=20749 timeouts=0 errors=0) read(tps=83104 timeouts=0 errors=0) total(tps=103853 timeouts=0 errors=0)
2021-04-02 16:27:27.328 write(tps=20548 timeouts=0 errors=0) read(tps=83258 timeouts=0 errors=0) total(tps=103806 timeouts=0 errors=0)
2021-04-02 16:27:28.328 write(tps=20826 timeouts=0 errors=0) read(tps=83232 timeouts=0 errors=0) total(tps=104058 timeouts=0 errors=0)
2021-04-02 16:27:29.328 write(tps=20539 timeouts=0 errors=0) read(tps=83561 timeouts=0 errors=0) total(tps=104100 timeouts=0 errors=0)
2021-04-02 16:27:30.329 write(tps=20774 timeouts=0 errors=0) read(tps=82985 timeouts=0 errors=0) total(tps=103759 timeouts=0 errors=0)
2021-04-02 16:27:31.329 write(tps=20752 timeouts=0 errors=0) read(tps=83179 timeouts=0 errors=0) total(tps=103931 timeouts=0 errors=0)
2021-04-02 16:27:32.329 write(tps=20780 timeouts=0 errors=0) read(tps=82848 timeouts=0 errors=0) total(tps=103628 timeouts=0 errors=0)
2021-04-02 16:27:33.329 write(tps=20680 timeouts=0 errors=0) read(tps=82824 timeouts=0 errors=0) total(tps=103504 timeouts=0 errors=0)
2021-04-02 16:27:34.330 write(tps=20800 timeouts=0 errors=0) read(tps=82640 timeouts=0 errors=0) total(tps=103440 timeouts=0 errors=0)
2021-04-02 16:27:35.330 write(tps=20612 timeouts=0 errors=0) read(tps=83023 timeouts=0 errors=0) total(tps=103635 timeouts=0 errors=0)
2021-04-02 16:27:36.330 write(tps=20715 timeouts=0 errors=0) read(tps=82478 timeouts=0 errors=0) total(tps=103193 timeouts=0 errors=0)
2021-04-02 16:27:37.330 write(tps=20604 timeouts=0 errors=0) read(tps=82733 timeouts=0 errors=0) total(tps=103337 timeouts=0 errors=0)
2021-04-02 16:27:38.331 write(tps=20577 timeouts=0 errors=0) read(tps=82778 timeouts=0 errors=0) total(tps=103355 timeouts=0 errors=0)
2021-04-02 16:27:39.331 write(tps=20721 timeouts=0 errors=0) read(tps=82211 timeouts=0 errors=0) total(tps=102932 timeouts=0 errors=0)
2021-04-02 16:27:40.331 write(tps=20377 timeouts=0 errors=0) read(tps=82687 timeouts=0 errors=0) total(tps=103064 timeouts=0 errors=0)
2021-04-02 16:27:41.332 write(tps=20525 timeouts=0 errors=0) read(tps=82198 timeouts=0 errors=0) total(tps=102723 timeouts=0 errors=0)
2021-04-02 16:27:42.332 write(tps=20648 timeouts=0 errors=0) read(tps=82022 timeouts=0 errors=0) total(tps=102670 timeouts=0 errors=0)
2021-04-02 16:27:43.332 write(tps=20456 timeouts=0 errors=0) read(tps=82493 timeouts=0 errors=0) total(tps=102949 timeouts=0 errors=0)
2021-04-02 16:27:44.333 write(tps=20586 timeouts=0 errors=0) read(tps=82869 timeouts=0 errors=0) total(tps=103455 timeouts=0 errors=0)
2021-04-02 16:27:45.333 write(tps=20591 timeouts=0 errors=0) read(tps=82082 timeouts=0 errors=0) total(tps=102673 timeouts=0 errors=0)
2021-04-02 16:27:46.333 write(tps=20441 timeouts=0 errors=0) read(tps=82666 timeouts=0 errors=0) total(tps=103107 timeouts=0 errors=0)
2021-04-02 16:27:47.333 write(tps=20639 timeouts=0 errors=0) read(tps=81723 timeouts=0 errors=0) total(tps=102362 timeouts=0 errors=0)
2021-04-02 16:27:48.334 write(tps=20464 timeouts=0 errors=0) read(tps=82241 timeouts=0 errors=0) total(tps=102705 timeouts=0 errors=0)
2021-04-02 16:27:49.334 write(tps=20434 timeouts=0 errors=0) read(tps=82254 timeouts=0 errors=0) total(tps=102688 timeouts=0 errors=0)
2021-04-02 16:27:50.334 write(tps=20344 timeouts=0 errors=0) read(tps=81824 timeouts=0 errors=0) total(tps=102168 timeouts=0 errors=0)
2021-04-02 16:27:51.334 write(tps=20386 timeouts=0 errors=0) read(tps=82027 timeouts=0 errors=0) total(tps=102413 timeouts=0 errors=0)
2021-04-02 16:27:52.335 write(tps=20476 timeouts=0 errors=0) read(tps=81761 timeouts=0 errors=0) total(tps=102237 timeouts=0 errors=0)
2021-04-02 16:27:53.335 write(tps=20473 timeouts=0 errors=0) read(tps=82109 timeouts=0 errors=0) total(tps=102582 timeouts=0 errors=0)
2021-04-02 16:27:54.335 write(tps=20623 timeouts=0 errors=0) read(tps=81575 timeouts=0 errors=0) total(tps=102198 timeouts=0 errors=0)
2021-04-02 16:27:55.335 write(tps=20403 timeouts=0 errors=0) read(tps=82009 timeouts=0 errors=0) total(tps=102412 timeouts=0 errors=0)
2021-04-02 16:27:56.336 write(tps=20307 timeouts=0 errors=0) read(tps=82104 timeouts=0 errors=0) total(tps=102411 timeouts=0 errors=0)
2021-04-02 16:27:57.336 write(tps=20551 timeouts=0 errors=0) read(tps=81385 timeouts=0 errors=0) total(tps=101936 timeouts=0 errors=0)
2021-04-02 16:27:58.336 write(tps=20422 timeouts=0 errors=0) read(tps=81847 timeouts=0 errors=0) total(tps=102269 timeouts=0 errors=0)
2021-04-02 16:27:59.336 write(tps=20590 timeouts=0 errors=0) read(tps=81347 timeouts=0 errors=0) total(tps=101937 timeouts=0 errors=0)
2021-04-02 16:28:00.336 write(tps=20203 timeouts=0 errors=0) read(tps=81682 timeouts=0 errors=0) total(tps=101885 timeouts=0 errors=0)
2021-04-02 16:28:01.337 write(tps=20367 timeouts=0 errors=0) read(tps=81765 timeouts=0 errors=0) total(tps=102132 timeouts=0 errors=0)
2021-04-02 16:28:02.337 write(tps=20384 timeouts=0 errors=0) read(tps=81582 timeouts=0 errors=0) total(tps=101966 timeouts=0 errors=0)
2021-04-02 16:28:03.337 write(tps=20224 timeouts=0 errors=0) read(tps=81330 timeouts=0 errors=0) total(tps=101554 timeouts=0 errors=0)
2021-04-02 16:28:04.337 write(tps=20352 timeouts=0 errors=0) read(tps=81512 timeouts=0 errors=0) total(tps=101864 timeouts=0 errors=0)
2021-04-02 16:28:05.337 write(tps=20349 timeouts=0 errors=0) read(tps=81452 timeouts=0 errors=0) total(tps=101801 timeouts=0 errors=0)
2021-04-02 16:28:06.338 write(tps=20308 timeouts=0 errors=0) read(tps=81128 timeouts=0 errors=0) total(tps=101436 timeouts=0 errors=0)
2021-04-02 16:28:07.339 write(tps=20351 timeouts=0 errors=0) read(tps=80948 timeouts=0 errors=0) total(tps=101299 timeouts=0 errors=0)

Why is this?

thanks

2

1- If you want to take a node down without (almost without) impacting the client, leverage the quiesce command. This is only available for the Enterprise Edition, though. You could also configure your timeout / retry for reads in a way that would force the read failures to retry against another copy and therefore minimize the impact (on reads at least).

2- Hard to say why the degradation. This would require checking the logs. Could be, for example due to defragmentation activity picking up at some point, or nsup starting or simply some slow down as the index size grows… but I wouldn’t expect it to be so drastic. Analyzing the logs and turning on specific read and write benchmarks may shed some light. (BTW, the total throughput seems to be in the ~100K across all 3 nodes on the server rather than 1M).

1 Like
3

Thank you very much for your reply. We have an enterprise license. I also need to test the quiesce function again.

According to my experience, it seems to be the same as you think. The query efficiency becomes low due to the increase of the index. Because after I clean up the data, I re-test the TPS and it will become higher again. Otherwise it will keep dropping.

In addition, the 1M I want to express refers to the total number of records in the namespace.

4

When the benchmark started, the database was empty, so all writes were inserts (nothing to update). As the test progressed, more of these inserts became updates (updates require a read). I suspect this is the primary cause of the perceived performance degradation.

You could confirm this by either running a linear insert load to initialize your key range before starting the above benchmark or configuring the benchmark to use ‘replace’ instead of ‘update’. A replace generally doesn’t need to read the record except when that record is a member of a secondary index.

1 Like
5

Hi @kporter

I have changed the workload mode from RU to RR, but the performance is still degraded.

Did you encounter this kind of problem during use? 
    
    [root@qa-aerospike-act aerospike]# timeout 120 bin/asbenchmark -h 10.114.1.210 -p 3000 -n test -k 10000000 -S 1 -o B:1024 -w RR,80 -z 20
Benchmark: 10.114.1.210 3000, namespace: test, set: testset, threads: 20, workload: READ_REPLACE
read: 80% (all bins: 100%, single bin: 0%), write: 20% (all bins: 100%, single bin: 0%)
keys: 10000000, start key: 1, transactions: 0, bins: 1, random values: false, throughput: unlimited
read policy:
    socketTimeout: 30000, totalTimeout: 0, timeoutDelay: 0, maxRetries: 2, sleepBetweenRetries: 0
    readModeAP: ONE, readModeSC: SESSION, replica: SEQUENCE, reportNotFound: false
write policy:
    socketTimeout: 30000, totalTimeout: 0, timeoutDelay: 0, maxRetries: 0, sleepBetweenRetries: 0
    commitLevel: COMMIT_ALL
Sync: connPoolsPerNode: 1
bin[0]: byte[1024]
debug: false
2021-04-07 10:14:16.346 INFO Thread main Add node BB97D9E363E1600 10.114.1.210 3000
2021-04-07 10:14:16.354 INFO Thread main Add node BB96C522E3E1600 10.114.1.211 3000
2021-04-07 10:14:16.354 INFO Thread main Add node BB93624143E1600 10.114.1.212 3000
2021-04-07 10:14:17.266 write(tps=14248 timeouts=0 errors=0) read(tps=58075 timeouts=0 errors=0) total(tps=72323 timeouts=0 errors=0)
2021-04-07 10:14:18.267 write(tps=25273 timeouts=0 errors=0) read(tps=102258 timeouts=0 errors=0) total(tps=127531 timeouts=0 errors=0)
2021-04-07 10:14:19.267 write(tps=26168 timeouts=0 errors=0) read(tps=105200 timeouts=0 errors=0) total(tps=131368 timeouts=0 errors=0)
2021-04-07 10:14:20.267 write(tps=26101 timeouts=0 errors=0) read(tps=104933 timeouts=0 errors=0) total(tps=131034 timeouts=0 errors=0)
2021-04-07 10:14:21.268 write(tps=26132 timeouts=0 errors=0) read(tps=104751 timeouts=0 errors=0) total(tps=130883 timeouts=0 errors=0)
2021-04-07 10:14:22.268 write(tps=26085 timeouts=0 errors=0) read(tps=103898 timeouts=0 errors=0) total(tps=129983 timeouts=0 errors=0)
2021-04-07 10:14:23.268 write(tps=25876 timeouts=0 errors=0) read(tps=103764 timeouts=0 errors=0) total(tps=129640 timeouts=0 errors=0)
2021-04-07 10:14:24.269 write(tps=26176 timeouts=0 errors=0) read(tps=104802 timeouts=0 errors=0) total(tps=130978 timeouts=0 errors=0)
2021-04-07 10:14:25.269 write(tps=26345 timeouts=0 errors=0) read(tps=104751 timeouts=0 errors=0) total(tps=131096 timeouts=0 errors=0)
2021-04-07 10:14:26.270 write(tps=26274 timeouts=0 errors=0) read(tps=104658 timeouts=0 errors=0) total(tps=130932 timeouts=0 errors=0)
2021-04-07 10:14:27.270 write(tps=26155 timeouts=0 errors=0) read(tps=105054 timeouts=0 errors=0) total(tps=131209 timeouts=0 errors=0)
2021-04-07 10:14:28.270 write(tps=26407 timeouts=0 errors=0) read(tps=104382 timeouts=0 errors=0) total(tps=130789 timeouts=0 errors=0)
2021-04-07 10:14:29.271 write(tps=26205 timeouts=0 errors=0) read(tps=104974 timeouts=0 errors=0) total(tps=131179 timeouts=0 errors=0)
2021-04-07 10:14:30.271 write(tps=26457 timeouts=0 errors=0) read(tps=104799 timeouts=0 errors=0) total(tps=131256 timeouts=0 errors=0)
2021-04-07 10:14:31.271 write(tps=26305 timeouts=0 errors=0) read(tps=104899 timeouts=0 errors=0) total(tps=131204 timeouts=0 errors=0)
2021-04-07 10:14:32.272 write(tps=26553 timeouts=0 errors=0) read(tps=104824 timeouts=0 errors=0) total(tps=131377 timeouts=0 errors=0)
2021-04-07 10:14:33.272 write(tps=26093 timeouts=0 errors=0) read(tps=105433 timeouts=0 errors=0) total(tps=131526 timeouts=0 errors=0)
2021-04-07 10:14:34.273 write(tps=26498 timeouts=0 errors=0) read(tps=104974 timeouts=0 errors=0) total(tps=131472 timeouts=0 errors=0)
2021-04-07 10:14:35.273 write(tps=26307 timeouts=0 errors=0) read(tps=105132 timeouts=0 errors=0) total(tps=131439 timeouts=0 errors=0)
2021-04-07 10:14:36.273 write(tps=26303 timeouts=0 errors=0) read(tps=104928 timeouts=0 errors=0) total(tps=131231 timeouts=0 errors=0)
2021-04-07 10:14:37.273 write(tps=26288 timeouts=0 errors=0) read(tps=105424 timeouts=0 errors=0) total(tps=131712 timeouts=0 errors=0)
2021-04-07 10:14:38.274 write(tps=26481 timeouts=0 errors=0) read(tps=104762 timeouts=0 errors=0) total(tps=131243 timeouts=0 errors=0)
2021-04-07 10:14:39.274 write(tps=26156 timeouts=0 errors=0) read(tps=105125 timeouts=0 errors=0) total(tps=131281 timeouts=0 errors=0)
2021-04-07 10:14:40.274 write(tps=26178 timeouts=0 errors=0) read(tps=104806 timeouts=0 errors=0) total(tps=130984 timeouts=0 errors=0)
2021-04-07 10:14:41.275 write(tps=26263 timeouts=0 errors=0) read(tps=104062 timeouts=0 errors=0) total(tps=130325 timeouts=0 errors=0)
2021-04-07 10:14:42.275 write(tps=26196 timeouts=0 errors=0) read(tps=105013 timeouts=0 errors=0) total(tps=131209 timeouts=0 errors=0)
2021-04-07 10:14:43.275 write(tps=26143 timeouts=0 errors=0) read(tps=104840 timeouts=0 errors=0) total(tps=130983 timeouts=0 errors=0)
2021-04-07 10:14:44.276 write(tps=26187 timeouts=0 errors=0) read(tps=105023 timeouts=0 errors=0) total(tps=131210 timeouts=0 errors=0)
2021-04-07 10:14:45.276 write(tps=26242 timeouts=0 errors=0) read(tps=104591 timeouts=0 errors=0) total(tps=130833 timeouts=0 errors=0)
2021-04-07 10:14:46.276 write(tps=26165 timeouts=0 errors=0) read(tps=104652 timeouts=0 errors=0) total(tps=130817 timeouts=0 errors=0)
2021-04-07 10:14:47.277 write(tps=26026 timeouts=0 errors=0) read(tps=105045 timeouts=0 errors=0) total(tps=131071 timeouts=0 errors=0)
2021-04-07 10:14:48.277 write(tps=26238 timeouts=0 errors=0) read(tps=104281 timeouts=0 errors=0) total(tps=130519 timeouts=0 errors=0)
2021-04-07 10:14:49.277 write(tps=26101 timeouts=0 errors=0) read(tps=104686 timeouts=0 errors=0) total(tps=130787 timeouts=0 errors=0)
2021-04-07 10:14:50.278 write(tps=26283 timeouts=0 errors=0) read(tps=104298 timeouts=0 errors=0) total(tps=130581 timeouts=0 errors=0)
2021-04-07 10:14:51.278 write(tps=25778 timeouts=0 errors=0) read(tps=103972 timeouts=0 errors=0) total(tps=129750 timeouts=0 errors=0)
2021-04-07 10:14:52.278 write(tps=26256 timeouts=0 errors=0) read(tps=104167 timeouts=0 errors=0) total(tps=130423 timeouts=0 errors=0)
2021-04-07 10:14:53.279 write(tps=26118 timeouts=0 errors=0) read(tps=104172 timeouts=0 errors=0) total(tps=130290 timeouts=0 errors=0)
2021-04-07 10:14:54.279 write(tps=26115 timeouts=0 errors=0) read(tps=103984 timeouts=0 errors=0) total(tps=130099 timeouts=0 errors=0)
2021-04-07 10:14:55.279 write(tps=26140 timeouts=0 errors=0) read(tps=103756 timeouts=0 errors=0) total(tps=129896 timeouts=0 errors=0)
2021-04-07 10:14:56.280 write(tps=25931 timeouts=0 errors=0) read(tps=103476 timeouts=0 errors=0) total(tps=129407 timeouts=0 errors=0)
2021-04-07 10:14:57.280 write(tps=26117 timeouts=0 errors=0) read(tps=103698 timeouts=0 errors=0) total(tps=129815 timeouts=0 errors=0)
2021-04-07 10:14:58.280 write(tps=26039 timeouts=0 errors=0) read(tps=103722 timeouts=0 errors=0) total(tps=129761 timeouts=0 errors=0)
2021-04-07 10:14:59.281 write(tps=26155 timeouts=0 errors=0) read(tps=103446 timeouts=0 errors=0) total(tps=129601 timeouts=0 errors=0)
2021-04-07 10:15:00.281 write(tps=26190 timeouts=0 errors=0) read(tps=103121 timeouts=0 errors=0) total(tps=129311 timeouts=0 errors=0)
2021-04-07 10:15:01.281 write(tps=25929 timeouts=0 errors=0) read(tps=102987 timeouts=0 errors=0) total(tps=128916 timeouts=0 errors=0)
2021-04-07 10:15:02.282 write(tps=25792 timeouts=0 errors=0) read(tps=103325 timeouts=0 errors=0) total(tps=129117 timeouts=0 errors=0)
2021-04-07 10:15:03.282 write(tps=25973 timeouts=0 errors=0) read(tps=103401 timeouts=0 errors=0) total(tps=129374 timeouts=0 errors=0)
2021-04-07 10:15:04.282 write(tps=25571 timeouts=0 errors=0) read(tps=104107 timeouts=0 errors=0) total(tps=129678 timeouts=0 errors=0)
2021-04-07 10:15:05.282 write(tps=25479 timeouts=0 errors=0) read(tps=103842 timeouts=0 errors=0) total(tps=129321 timeouts=0 errors=0)
2021-04-07 10:15:06.283 write(tps=25857 timeouts=0 errors=0) read(tps=102733 timeouts=0 errors=0) total(tps=128590 timeouts=0 errors=0)
2021-04-07 10:15:07.283 write(tps=25560 timeouts=0 errors=0) read(tps=103232 timeouts=0 errors=0) total(tps=128792 timeouts=0 errors=0)
2021-04-07 10:15:08.283 write(tps=25936 timeouts=0 errors=0) read(tps=102765 timeouts=0 errors=0) total(tps=128701 timeouts=0 errors=0)
2021-04-07 10:15:09.284 write(tps=25783 timeouts=0 errors=0) read(tps=102970 timeouts=0 errors=0) total(tps=128753 timeouts=0 errors=0)
2021-04-07 10:15:10.284 write(tps=25720 timeouts=0 errors=0) read(tps=102684 timeouts=0 errors=0) total(tps=128404 timeouts=0 errors=0)
2021-04-07 10:15:11.284 write(tps=25386 timeouts=0 errors=0) read(tps=101744 timeouts=0 errors=0) total(tps=127130 timeouts=0 errors=0)
2021-04-07 10:15:12.285 write(tps=25634 timeouts=0 errors=0) read(tps=101850 timeouts=0 errors=0) total(tps=127484 timeouts=0 errors=0)
2021-04-07 10:15:13.285 write(tps=25573 timeouts=0 errors=0) read(tps=102771 timeouts=0 errors=0) total(tps=128344 timeouts=0 errors=0)
2021-04-07 10:15:14.285 write(tps=25653 timeouts=0 errors=0) read(tps=102801 timeouts=0 errors=0) total(tps=128454 timeouts=0 errors=0)
2021-04-07 10:15:15.285 write(tps=25769 timeouts=0 errors=0) read(tps=102030 timeouts=0 errors=0) total(tps=127799 timeouts=0 errors=0)
2021-04-07 10:15:16.286 write(tps=25744 timeouts=0 errors=0) read(tps=102863 timeouts=0 errors=0) total(tps=128607 timeouts=0 errors=0)
2021-04-07 10:15:17.286 write(tps=25384 timeouts=0 errors=0) read(tps=102510 timeouts=0 errors=0) total(tps=127894 timeouts=0 errors=0)
2021-04-07 10:15:18.286 write(tps=25441 timeouts=0 errors=0) read(tps=102256 timeouts=0 errors=0) total(tps=127697 timeouts=0 errors=0)
2021-04-07 10:15:19.287 write(tps=25583 timeouts=0 errors=0) read(tps=102609 timeouts=0 errors=0) total(tps=128192 timeouts=0 errors=0)
2021-04-07 10:15:20.287 write(tps=25742 timeouts=0 errors=0) read(tps=101883 timeouts=0 errors=0) total(tps=127625 timeouts=0 errors=0)
2021-04-07 10:15:21.287 write(tps=25442 timeouts=0 errors=0) read(tps=102178 timeouts=0 errors=0) total(tps=127620 timeouts=0 errors=0)
2021-04-07 10:15:22.287 write(tps=25585 timeouts=0 errors=0) read(tps=101935 timeouts=0 errors=0) total(tps=127520 timeouts=0 errors=0)
2021-04-07 10:15:23.288 write(tps=25299 timeouts=0 errors=0) read(tps=102456 timeouts=0 errors=0) total(tps=127755 timeouts=0 errors=0)
2021-04-07 10:15:24.288 write(tps=25580 timeouts=0 errors=0) read(tps=102181 timeouts=0 errors=0) total(tps=127761 timeouts=0 errors=0)
2021-04-07 10:15:25.288 write(tps=25422 timeouts=0 errors=0) read(tps=102098 timeouts=0 errors=0) total(tps=127520 timeouts=0 errors=0)
2021-04-07 10:15:26.288 write(tps=25476 timeouts=0 errors=0) read(tps=101415 timeouts=0 errors=0) total(tps=126891 timeouts=0 errors=0)
2021-04-07 10:15:27.289 write(tps=25325 timeouts=0 errors=0) read(tps=102081 timeouts=0 errors=0) total(tps=127406 timeouts=0 errors=0)
2021-04-07 10:15:28.289 write(tps=25272 timeouts=0 errors=0) read(tps=102103 timeouts=0 errors=0) total(tps=127375 timeouts=0 errors=0)
2021-04-07 10:15:29.289 write(tps=25495 timeouts=0 errors=0) read(tps=101226 timeouts=0 errors=0) total(tps=126721 timeouts=0 errors=0)
2021-04-07 10:15:30.290 write(tps=25419 timeouts=0 errors=0) read(tps=101310 timeouts=0 errors=0) total(tps=126729 timeouts=0 errors=0)
2021-04-07 10:15:31.290 write(tps=25079 timeouts=0 errors=0) read(tps=101377 timeouts=0 errors=0) total(tps=126456 timeouts=0 errors=0)
2021-04-07 10:15:32.290 write(tps=25355 timeouts=0 errors=0) read(tps=101343 timeouts=0 errors=0) total(tps=126698 timeouts=0 errors=0)
2021-04-07 10:15:33.291 write(tps=25408 timeouts=0 errors=0) read(tps=101583 timeouts=0 errors=0) total(tps=126991 timeouts=0 errors=0)
2021-04-07 10:15:34.291 write(tps=25398 timeouts=0 errors=0) read(tps=100881 timeouts=0 errors=0) total(tps=126279 timeouts=0 errors=0)
2021-04-07 10:15:35.291 write(tps=25431 timeouts=0 errors=0) read(tps=100886 timeouts=0 errors=0) total(tps=126317 timeouts=0 errors=0)
2021-04-07 10:15:36.292 write(tps=25152 timeouts=0 errors=0) read(tps=101423 timeouts=0 errors=0) total(tps=126575 timeouts=0 errors=0)
2021-04-07 10:15:37.292 write(tps=25237 timeouts=0 errors=0) read(tps=101200 timeouts=0 errors=0) total(tps=126437 timeouts=0 errors=0)
2021-04-07 10:15:38.297 write(tps=25399 timeouts=0 errors=0) read(tps=101637 timeouts=0 errors=0) total(tps=127036 timeouts=0 errors=0)
2021-04-07 10:15:39.297 write(tps=25180 timeouts=0 errors=0) read(tps=101082 timeouts=0 errors=0) total(tps=126262 timeouts=0 errors=0)
2021-04-07 10:15:40.297 write(tps=25082 timeouts=0 errors=0) read(tps=101073 timeouts=0 errors=0) total(tps=126155 timeouts=0 errors=0)
2021-04-07 10:15:41.298 write(tps=25001 timeouts=0 errors=0) read(tps=100465 timeouts=0 errors=0) total(tps=125466 timeouts=0 errors=0)
2021-04-07 10:15:42.298 write(tps=25058 timeouts=0 errors=0) read(tps=101117 timeouts=0 errors=0) total(tps=126175 timeouts=0 errors=0)
2021-04-07 10:15:43.298 write(tps=25145 timeouts=0 errors=0) read(tps=100865 timeouts=0 errors=0) total(tps=126010 timeouts=0 errors=0)
2021-04-07 10:15:44.298 write(tps=25165 timeouts=0 errors=0) read(tps=100658 timeouts=0 errors=0) total(tps=125823 timeouts=0 errors=0)
2021-04-07 10:15:45.299 write(tps=24963 timeouts=0 errors=0) read(tps=100916 timeouts=0 errors=0) total(tps=125879 timeouts=0 errors=0)
2021-04-07 10:15:46.299 write(tps=25142 timeouts=0 errors=0) read(tps=100784 timeouts=0 errors=0) total(tps=125926 timeouts=0 errors=0)
2021-04-07 10:15:47.299 write(tps=24925 timeouts=0 errors=0) read(tps=100866 timeouts=0 errors=0) total(tps=125791 timeouts=0 errors=0)
2021-04-07 10:15:48.299 write(tps=25182 timeouts=0 errors=0) read(tps=100374 timeouts=0 errors=0) total(tps=125556 timeouts=0 errors=0)
2021-04-07 10:15:49.300 write(tps=25042 timeouts=0 errors=0) read(tps=100809 timeouts=0 errors=0) total(tps=125851 timeouts=0 errors=0)
2021-04-07 10:15:50.300 write(tps=25015 timeouts=0 errors=0) read(tps=100461 timeouts=0 errors=0) total(tps=125476 timeouts=0 errors=0)
2021-04-07 10:15:51.300 write(tps=24894 timeouts=0 errors=0) read(tps=99508 timeouts=0 errors=0) total(tps=124402 timeouts=0 errors=0)
2021-04-07 10:15:52.301 write(tps=25130 timeouts=0 errors=0) read(tps=100296 timeouts=0 errors=0) total(tps=125426 timeouts=0 errors=0)
2021-04-07 10:15:53.301 write(tps=24922 timeouts=0 errors=0) read(tps=100279 timeouts=0 errors=0) total(tps=125201 timeouts=0 errors=0)
2021-04-07 10:15:54.301 write(tps=25292 timeouts=0 errors=0) read(tps=99852 timeouts=0 errors=0) total(tps=125144 timeouts=0 errors=0)
2021-04-07 10:15:55.301 write(tps=24997 timeouts=0 errors=0) read(tps=99630 timeouts=0 errors=0) total(tps=124627 timeouts=0 errors=0)
2021-04-07 10:15:56.302 write(tps=24805 timeouts=0 errors=0) read(tps=99948 timeouts=0 errors=0) total(tps=124753 timeouts=0 errors=0)
2021-04-07 10:15:57.302 write(tps=25002 timeouts=0 errors=0) read(tps=100230 timeouts=0 errors=0) total(tps=125232 timeouts=0 errors=0)
2021-04-07 10:15:58.302 write(tps=24937 timeouts=0 errors=0) read(tps=100222 timeouts=0 errors=0) total(tps=125159 timeouts=0 errors=0)
2021-04-07 10:15:59.302 write(tps=24989 timeouts=0 errors=0) read(tps=100140 timeouts=0 errors=0) total(tps=125129 timeouts=0 errors=0)
2021-04-07 10:16:00.303 write(tps=24686 timeouts=0 errors=0) read(tps=100505 timeouts=0 errors=0) total(tps=125191 timeouts=0 errors=0)
2021-04-07 10:16:01.303 write(tps=24907 timeouts=0 errors=0) read(tps=99455 timeouts=0 errors=0) total(tps=124362 timeouts=0 errors=0)
2021-04-07 10:16:02.303 write(tps=24873 timeouts=0 errors=0) read(tps=100186 timeouts=0 errors=0) total(tps=125059 timeouts=0 errors=0)
2021-04-07 10:16:03.303 write(tps=24890 timeouts=0 errors=0) read(tps=100160 timeouts=0 errors=0) total(tps=125050 timeouts=0 errors=0)
2021-04-07 10:16:04.304 write(tps=24990 timeouts=0 errors=0) read(tps=99534 timeouts=0 errors=0) total(tps=124524 timeouts=0 errors=0)
2021-04-07 10:16:05.304 write(tps=25319 timeouts=0 errors=0) read(tps=98978 timeouts=0 errors=0) total(tps=124297 timeouts=0 errors=0)
2021-04-07 10:16:06.304 write(tps=24931 timeouts=0 errors=0) read(tps=99552 timeouts=0 errors=0) total(tps=124483 timeouts=0 errors=0)
2021-04-07 10:16:07.304 write(tps=25072 timeouts=0 errors=0) read(tps=98826 timeouts=0 errors=0) total(tps=123898 timeouts=0 errors=0)
2021-04-07 10:16:08.305 write(tps=24878 timeouts=0 errors=0) read(tps=99544 timeouts=0 errors=0) total(tps=124422 timeouts=0 errors=0)
2021-04-07 10:16:09.305 write(tps=24704 timeouts=0 errors=0) read(tps=99683 timeouts=0 errors=0) total(tps=124387 timeouts=0 errors=0)
2021-04-07 10:16:10.305 write(tps=24830 timeouts=0 errors=0) read(tps=99438 timeouts=0 errors=0) total(tps=124268 timeouts=0 errors=0)
2021-04-07 10:16:11.305 write(tps=24600 timeouts=0 errors=0) read(tps=98926 timeouts=0 errors=0) total(tps=123526 timeouts=0 errors=0)
2021-04-07 10:16:12.306 write(tps=24995 timeouts=0 errors=0) read(tps=98988 timeouts=0 errors=0) total(tps=123983 timeouts=0 errors=0)
2021-04-07 10:16:13.306 write(tps=24897 timeouts=0 errors=0) read(tps=99465 timeouts=0 errors=0) total(tps=124362 timeouts=0 errors=0)
2021-04-07 10:16:14.306 write(tps=24909 timeouts=0 errors=0) read(tps=99215 timeouts=0 errors=0) total(tps=124124 timeouts=0 errors=0)
2021-04-07 10:16:15.306 write(tps=24670 timeouts=0 errors=0) read(tps=99549 timeouts=0 errors=0) total(tps=124219 timeouts=0 errors=0)
6

Yes, there is still some degradation expected, but, as expected, the magnitude of that degradation is dramatically reduced by using replace. In your the first benchmark, the final ‘total tps’ was ~82% of the peak ‘total tps’ while in the second benchmark, the final ‘total tps’ was ~95% of the peak ‘total tps’. As @meher already pointed out, the next source of degradation is likely going to come from defrag. As this benchmark progresses, write-blocks become eligible for defragmentation as updates, replaces, or deletes cause records to vacate blocks the records previously resided in. With the default defrag-lwm-pct the equilibrium write-amplification caused by defrag is 2x.

7

BTW, our Aerospike Certification Tool (ACT) is designed to simulate Aerospike in equilibrium. You can use this tool to get an idea of what sort of performance you can expect from your devices. You should also note that we recommend running ACT for several hours because the performance of the devices themselves eventually degrade due to device firmware level processes such as write-leveling and defragmentation.