# Where is the bottleneck? Running Aerospike on NVMe **URL:** \ **Category:** Aerospike Server Benchmarks\ **Created:** [April 11, 2018, 9:09pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103 "2018-04-11T21:09:42Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![hedayati](https://avatars.discourse-cdn.com/v4/letter/h/a88e57/32.png) [@hedayati](https://discuss.aerospike.com/u/hedayati)\ **Post date:** [April 11, 2018, 9:09pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/1 "2018-04-11T21:09:42Z")
I am trying to run a local instance of aerospike server on a 2 x E5-2630 v3 (32 HT) machine. The attached nvme-of device (`/dev/nvem0n1`) is capable of over 2M IOPs for 1K IOs reads – verified by FIO (I am using `B:100` in my benchmarks, only reads). ``` service { paxos-single-replica-limit 1 work-directory run/work pidfile run/asd.pid service-threads 16 transaction-queues 8 transaction-threads-per-queue 8 proto-fd-max 100000 } namespace test { replication-factor 2 memory-size 1G default-ttl 30d # 30 days, use 0 to never expire/evict. storage-engine device { # Use one or more lines like those below with actual device paths. device /dev/nvme0n1 # The 2 lines below optimize for SSD. scheduler-mode noop write-block-size 4K data-in-memory false } } ``` I tried running the benchmark both locally and on a different node. In both cases, I get less than 1M TPS (local: ~700K, remote: ~600K). ``` PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6441 root 20 0 4285896 290072 6272 S 2051 0.4 413:54.88 asd 6981 root 20 0 4853380 9612 4260 S 1088 0.0 15:29.61 benchmarks ``` And this is how `perf top -p asdpid` looks like. ``` 6.82% [kernel] [k] queued_spin_lock_slowpath 1.60% [kernel] [k] _raw_spin_lock_irqsave 1.49% [kernel] [k] tcp_ack 1.47% [kernel] [k] __fget 1.23% libc-2.21.so [.] __memcmp_sse4_1 1.18% [kernel] [k] _raw_spin_lock 1.17% libpthread-2.21.so [.] pthread_mutex_lock 1.07% [kernel] [k] try_to_wake_up 1.01% asd [.] as_index_get_v ``` Edited for clarification: I am not running on a real SSD. The benchmark is running on a NVMe-OF device backed by DRAM on the target side (The bandwidth is limited by RDMA over IB link and the latencies are in O(10usec)). ---
**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\ **Post date:** [April 11, 2018, 9:46pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/2 "2018-04-11T21:46:50Z")
What do the histograms look like? ---
**Author:** ![hedayati](https://avatars.discourse-cdn.com/v4/letter/h/a88e57/32.png) [@hedayati](https://discuss.aerospike.com/u/hedayati)\ **Post date:** [April 12, 2018, 2:30pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/3 "2018-04-12T14:30:40Z")
``` Apr 12 2018 14:11:08 GMT: INFO (info): (hist.c:139) histogram dump: {test}-read (36165690641 total) msec Apr 12 2018 14:11:08 GMT: INFO (info): (hist.c:156) (00: 22963278903) (01: 5763021768) (02: 3246872646) (03: 2691991850) Apr 12 2018 14:11:08 GMT: INFO (info): (hist.c:165) (04: 1500507183) (05: 0000018057) (06: 0000000234) ``` Is this what you are asking for? ---
**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\ **Post date:** [April 12, 2018, 3:07pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/4 "2018-04-12T15:07:06Z")
32 Hyperthreads/1K objects - can you try service-threads =32, transaction-queues=32 and transaction-threads-per-queue=3? ---
**Author:** ![hedayati](https://avatars.discourse-cdn.com/v4/letter/h/a88e57/32.png) [@hedayati](https://discuss.aerospike.com/u/hedayati)\ **Post date:** [April 12, 2018, 3:31pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/5 "2018-04-12T15:31:26Z")
Doesn’t seem to help (no significant change in TPS). An additional data point that may help is that when I set `data-in-memory true` I can get 1.2M IOPs and I don’t see the `queued_spin_lock_slowpath` in `perf top` results. So whatever the bottleneck is, it has to be related to Flash (either some part in Aerospike on Linux storage stack). I am using the [https://github.com/aerospike/aerospike-server@8f0bb73](https://github.com/aerospike/aerospike-server/commit/8f0bb732a2d2bfd96a048a2175b092a81dd36670). ---
**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\ **Post date:** [April 12, 2018, 3:47pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/6 "2018-04-12T15:47:31Z")
Per this discussion thread, [FAQ - What is the purpose of setting the disk scheduler?](https://discuss.aerospike.com/t/faq-what-is-the-purpose-of-setting-the-disk-scheduler/3680) can you try scheduler-mode none? ---
**Author:** ![hedayati](https://avatars.discourse-cdn.com/v4/letter/h/a88e57/32.png) [@hedayati](https://discuss.aerospike.com/u/hedayati)\ **Post date:** [April 12, 2018, 4:57pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/7 "2018-04-12T16:57:22Z")
Still no change after setting the scheduler to `none`. ---
**Author:** ![hedayati](https://avatars.discourse-cdn.com/v4/letter/h/a88e57/32.png) [@hedayati](https://discuss.aerospike.com/u/hedayati)\ **Post date:** [April 12, 2018, 7:40pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/8 "2018-04-12T19:40:45Z")
After a little more profiling, it seems like the `cf_queue_pop` and `cf_queue_push` in [`ssd_fd_get()`](https://github.com/aerospike/aerospike-server/blob/master/as/src/storage/drv_ssd.c#L90) and [`ssd_fd_put()`](https://github.com/aerospike/aerospike-server/blob/master/as/src/storage/drv_ssd.c#L129) are not scalable enough to handle the load. Any thoughts how I may get around that? (Since I couldn’t find anyone else having similar issues, I tend to think there should be fix by changing the configurations, e.g., # of threads somewhere?) ---
**Author:** ![rbotzer](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rbotzer/32/2114_2.png) [@rbotzer](https://discuss.aerospike.com/u/rbotzer)\ **Post date:** [April 12, 2018, 8:50pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/9 "2018-04-12T20:50:25Z")
I’m going to question this assertion: " 2M IOPs for 1K IOs reads – verified by FIO". FIO is not able to generate the type of workloads that predict what the sustainable performance is for an SSD used by a database. How long did your FIO test run? SSDs have all sorts of caching tricks, from over-provisioning to others. A test of the SSD should run for enough hours at a sustained rate to truly understand what you can expect of it. For this reason, Aerospike created the [ACT](https://github.com/aerospike/act) tool, as a way to benchmark SSDs under sustained database workloads (in particular ones that simulate Aerospike). You haven’t mentioned which SSD you’re using, so it would be good for you to run ACT and confirm the drive capability before moving on to using asbenchmark (included in the [latest tools package](https://www.aerospike.com/download/tools/3.15.3.2/)). ---
**Author:** ![hedayati](https://avatars.discourse-cdn.com/v4/letter/h/a88e57/32.png) [@hedayati](https://discuss.aerospike.com/u/hedayati)\ **Post date:** [April 12, 2018, 9:22pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/10 "2018-04-12T21:22:18Z")
I am not running on a real SSD. The benchmark is running on a NVMe-OF device backed by DRAM on the target side (The bandwidth is limited by RDMA over IB link and the latencies are in O(10usec)). Thanks for the ACT pointer. Will take a look and report back. ---
**Author:** ![hedayati](https://avatars.discourse-cdn.com/v4/letter/h/a88e57/32.png) [@hedayati](https://discuss.aerospike.com/u/hedayati)\ **Post date:** [April 13, 2018, 1:59pm UTC](https://discuss.aerospike.com/t/where-is-the-bottleneck-running-aerospike-on-nvme/5103/11 "2018-04-13T13:59:11Z")
I created per-thread pool of open file descriptors for `ssd_fd_get()` and `ssd_fd_put()` and now the lock bottleneck is gone. With this change, I can get around 900 KIOPs before occupying all 32 hyperthreads. Does these number make sense? How many IOPs would you expect to get with this number of HTs on a fast SSD?