# Zig Zag transaction graph while perf testing Aerospike on ssd with 2 boxes

**URL:** <https://discuss.aerospike.com/t/zig-zag-transaction-graph-while-perf-testing-aerospike-on-ssd-with-2-boxes/8122>\
**Category:** Operations\
**Created:** [January 1, 2021, 1:14pm UTC](https://discuss.aerospike.com/t/zig-zag-transaction-graph-while-perf-testing-aerospike-on-ssd-with-2-boxes/8122 "2021-01-01T13:14:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saksham\_Agrawal](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/saksham_agrawal/32/1592_2.png) [@Saksham\_Agrawal](https://discuss.aerospike.com/u/Saksham_Agrawal)\
**Post date:** [January 1, 2021, 1:14pm UTC](https://discuss.aerospike.com/t/zig-zag-transaction-graph-while-perf-testing-aerospike-on-ssd-with-2-boxes/8122/1 "2021-01-01T13:14:05Z")

</div>

We are trying to benchmark aerospike and we are getting a zig-zag graph of TPS with benchmark tool Box: 20 cores, SSD. GRAPH:

 ![Screenshot 2021-01-01 at 6.40.47 PM](https://us1.discourse-cdn.com/flex019/uploads/aerospike/original/2X/2/2e351d213476257a97bd7d0f130671b3fb2f8a26.png)

```auto
Config:
service {
	user root
	group root
    pidfile /var/run/aerospike/asd.pid
}

logging {
        file /var/log/aerospike/aerospike.log {
		context any info
		context migrate debug
        }
}

network {
	service {
		address any
		port 3000
	}
	heartbeat {
		mode mesh
		port 3002
		mesh-seed-address-port 10.34.145.92	3002
		mesh-seed-address-port 10.32.78.156 3002

		interval 150
		timeout 10
	}

	fabric {
		port 3001
	}

	info {
		port 3003
	}
}

namespace testDisk {
	replication-factor 2
	memory-size 40G
	storage-engine device {
		file /opt/aerospike/data/entities.dat
		filesize 200G
		max-write-cache 128M	
		write-block-size 512K
		scheduler-mode noop
	}
}

```

Tried with decreasing the block size to 128K then we get error of **queue-too deep**. So increased the max-write-cache size as a workaround.

Still we are not able to get stable a TPS.

benchMark Command: ./run\_benmarks -h 10.34.145.92 -p 3000 -n testDisk -k 40000000 -S 1 -o S:4096 -w RU,0 -z 2 -async -asyncMaxCommands 50 -eventLoops 10 -latency ycsb

Please help us in narrowing down the issue.

---

<div class="post-metadata">

**Author:** ![meher](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/meher/32/1644_2.png) [@meher](https://discuss.aerospike.com/u/meher)\
**Post date:** [January 5, 2021, 1:59am UTC](https://discuss.aerospike.com/t/zig-zag-transaction-graph-while-perf-testing-aerospike-on-ssd-with-2-boxes/8122/2 "2021-01-05T01:59:49Z")

</div>

It looks like your node is not able to keep up… increasing `max-write-cache` would just ‘hide’ the errors but is an indication the storage subsystem is not keeping up. Try capping the throughput to get to a point where you can sustain the workload.

Reducing the `write-block-size` would end up reducing the `post-write-queue`… so you can ‘help’ by increasing the `post-write-queue` to help the storage sub-system and reducing the throughput.

---

<div class="post-metadata">

**Author:** ![Saksham\_Agrawal](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/saksham_agrawal/32/1592_2.png) [@Saksham\_Agrawal](https://discuss.aerospike.com/u/Saksham_Agrawal)\
**Post date:** [January 5, 2021, 9:38am UTC](https://discuss.aerospike.com/t/zig-zag-transaction-graph-while-perf-testing-aerospike-on-ssd-with-2-boxes/8122/3 "2021-01-05T09:38:36Z")

</div>

We are using 3.12 version and we did make post-write-queue as 2048(maximum), still it was not able to stabilise. Also we are using async client for benchmarking and in that case how can we reduce throughput ? [AFAIK we can give concurrency parameters but throughput might be governed by the latency.]

Also, we noticed a pattern that as soon as read disk ops come into picture that is when we start seeing drops and then that mountain pattern continues.

UPDATE: The issue was happening due to wrong configuration recipe, we changed the above configuration to device instead of file and this seems to work well.

---

<div class="post-metadata">

**Author:** ![meher](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/meher/32/1644_2.png) [@meher](https://discuss.aerospike.com/u/meher)\
**Post date:** [January 8, 2021, 4:36am UTC](https://discuss.aerospike.com/t/zig-zag-transaction-graph-while-perf-testing-aerospike-on-ssd-with-2-boxes/8122/4 "2021-01-08T04:36:07Z")

</div>

Thanks for the update. Here is [the doc](https://www.aerospike.com/docs/tools/asbenchmark/index.html) for the benchmark tool. Specifically the `-g` config for limiting throughput and `-z` for controlling the number of threads.

Check the [read-page-cache](https://www.aerospike.com/docs/reference/configuration/?show-removed=1#read-page-cache) parameter as well. The `file` config would have given you some of that and I am surprised it is making such a difference for you… maybe if you are a bit limited in RAM and the page cache churn is hurting at some point? In which case having `device` rather than `file` would give more consistent/predictable performance.

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/system/32/2274_2.png) [@system](https://discuss.aerospike.com/u/system)\
**Post date:** [January 8, 2022, 4:36am UTC](https://discuss.aerospike.com/t/zig-zag-transaction-graph-while-perf-testing-aerospike-on-ssd-with-2-boxes/8122/5 "2022-01-08T04:36:09Z")

</div>

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.