SSD storage engine slowdown?

Operations Tuning
1

I config SDD for storage engine and run the benchmark with Go client first run:

quanlm@quanlm2:~/go/src/github.com/aerospike/aerospike-client-go/tools/benchmark$ ./benchmark -w I -c 50
2019/12/17 11:31:01 Setting number of CPUs to use: 12
2019/12/17 11:31:01 benchmark.go:197: hosts:		127.0.0.1
2019/12/17 11:31:01 benchmark.go:198: port:		3000
2019/12/17 11:31:01 benchmark.go:199: namespace:	test
2019/12/17 11:31:01 benchmark.go:200: set:		testset
2019/12/17 11:31:01 benchmark.go:201: keys/records:	1000000
2019/12/17 11:31:01 benchmark.go:202: object spec:	I, size: 0
2019/12/17 11:31:01 benchmark.go:203: random bin values:	false
2019/12/17 11:31:01 benchmark.go:204: workload:		Initialize 100% of records
2019/12/17 11:31:01 benchmark.go:205: concurrency:	50
2019/12/17 11:31:01 benchmark.go:206: max throughput:	unlimited
2019/12/17 11:31:01 benchmark.go:207: timeout:		0 ms
2019/12/17 11:31:01 benchmark.go:208: max retries:	2
2019/12/17 11:31:01 benchmark.go:209: debug:		false
2019/12/17 11:31:01 benchmark.go:210: latency:		0:0
2019/12/17 11:31:01 benchmark.go:211: conn. pool size:	128
2019/12/17 11:31:01 benchmark.go:212: conn. threshold:	64
2019/12/17 11:31:01 benchmark.go:151: Warm-up conns.	:120
2019/12/17 11:31:01 benchmark.go:153: Nodes Found: [BB96A0BE73E16FA]
2019/12/17 11:31:02 benchmark.go:730: write(tps=184922 timeouts=0 errors=0 totalCount=184922)
2019/12/17 11:31:03 benchmark.go:730: write(tps=202971 timeouts=0 errors=0 totalCount=387893)
2019/12/17 11:31:04 benchmark.go:730: write(tps=202793 timeouts=0 errors=0 totalCount=590686)
2019/12/17 11:31:05 benchmark.go:730: write(tps=189889 timeouts=0 errors=0 totalCount=780575)
2019/12/17 11:31:06 benchmark.go:730: write(tps=205561 timeouts=0 errors=0 totalCount=986136)
2019/12/17 11:31:07 benchmark.go:730: write(tps=13864 timeouts=0 errors=0 totalCount=1000000)

But 2nd run

quanlm@quanlm2:~/go/src/github.com/aerospike/aerospike-client-go/tools/benchmark$ ./benchmark -w I -c 50
2019/12/17 11:31:21 Setting number of CPUs to use: 12
2019/12/17 11:31:21 benchmark.go:197: hosts:		127.0.0.1
2019/12/17 11:31:21 benchmark.go:198: port:		3000
2019/12/17 11:31:21 benchmark.go:199: namespace:	test
2019/12/17 11:31:21 benchmark.go:200: set:		testset
2019/12/17 11:31:21 benchmark.go:201: keys/records:	1000000
2019/12/17 11:31:21 benchmark.go:202: object spec:	I, size: 0
2019/12/17 11:31:21 benchmark.go:203: random bin values:	false
2019/12/17 11:31:21 benchmark.go:204: workload:		Initialize 100% of records
2019/12/17 11:31:21 benchmark.go:205: concurrency:	50
2019/12/17 11:31:21 benchmark.go:206: max throughput:	unlimited
2019/12/17 11:31:21 benchmark.go:207: timeout:		0 ms
2019/12/17 11:31:21 benchmark.go:208: max retries:	2
2019/12/17 11:31:21 benchmark.go:209: debug:		false
2019/12/17 11:31:21 benchmark.go:210: latency:		0:0
2019/12/17 11:31:21 benchmark.go:211: conn. pool size:	128
2019/12/17 11:31:21 benchmark.go:212: conn. threshold:	64
2019/12/17 11:31:21 benchmark.go:151: Warm-up conns.	:120
2019/12/17 11:31:21 benchmark.go:153: Nodes Found: [BB96A0BE73E16FA]
2019/12/17 11:31:22 benchmark.go:730: write(tps=557 timeouts=0 errors=0 totalCount=557)
2019/12/17 11:31:23 benchmark.go:730: write(tps=395 timeouts=0 errors=0 totalCount=952)

I have hard clean the disk using dd command No file system, just raw disk

My config:

namespace test {
        replication-factor 2
        memory-size 2G
        default-ttl 30d # 30 days, use 0 to never expire/evict.

        storage-engine device {
            device /dev/vdb
            write-block-size 128K
        }
}
2

Would be helpful if you included the parameters you use to invoke the go benchmark?

The first run, you were creating all of the records. The second run you are updating the records. The performance characteristics of create vs update are very different. A create just buffers the write to be written to disk. An update has to read the existing copy before buffering the new version to the disk.

If the go client supported a ‘replace’ workload option, you would see similar performance to the create workload because replace simply replaces the current record with the bins sent in the write without reading and merging with the old bins.

In the java benchmark, you would run:

./run_benchmarks -h 127.0.0.1 -w RR,0 -z 50 -n test -s testset -S 0 -k 1000000