I am trying to run a local instance of aerospike server on a 2 x E5-2630 v3 (32 HT) machine. The attached nvme-of device (/dev/nvem0n1) is capable of over 2M IOPs for 1K IOs reads – verified by FIO (I am using B:100 in my benchmarks, only reads).
service {
paxos-single-replica-limit 1
work-directory run/work
pidfile run/asd.pid
service-threads 16
transaction-queues 8
transaction-threads-per-queue 8
proto-fd-max 100000
}
namespace test {
replication-factor 2
memory-size 1G
default-ttl 30d # 30 days, use 0 to never expire/evict.
storage-engine device {
# Use one or more lines like those below with actual device paths.
device /dev/nvme0n1
# The 2 lines below optimize for SSD.
scheduler-mode noop
write-block-size 4K
data-in-memory false
}
}
I tried running the benchmark both locally and on a different node. In both cases, I get less than 1M TPS (local: ~700K, remote: ~600K).
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6441 root 20 0 4285896 290072 6272 S 2051 0.4 413:54.88 asd
6981 root 20 0 4853380 9612 4260 S 1088 0.0 15:29.61 benchmarks
Edited for clarification:
I am not running on a real SSD. The benchmark is running on a NVMe-OF device backed by DRAM on the target side (The bandwidth is limited by RDMA over IB link and the latencies are in O(10usec)).
Doesn’t seem to help (no significant change in TPS).
An additional data point that may help is that when I set data-in-memory true I can get 1.2M IOPs and I don’t see the queued_spin_lock_slowpath in perf top results. So whatever the bottleneck is, it has to be related to Flash (either some part in Aerospike on Linux storage stack).
After a little more profiling, it seems like the cf_queue_pop and cf_queue_push in ssd_fd_get() and ssd_fd_put() are not scalable enough to handle the load. Any thoughts how I may get around that? (Since I couldn’t find anyone else having similar issues, I tend to think there should be fix by changing the configurations, e.g., # of threads somewhere?)
I’m going to question this assertion: " 2M IOPs for 1K IOs reads – verified by FIO". FIO is not able to generate the type of workloads that predict what the sustainable performance is for an SSD used by a database. How long did your FIO test run? SSDs have all sorts of caching tricks, from over-provisioning to others. A test of the SSD should run for enough hours at a sustained rate to truly understand what you can expect of it.
For this reason, Aerospike created the ACT tool, as a way to benchmark SSDs under sustained database workloads (in particular ones that simulate Aerospike). You haven’t mentioned which SSD you’re using, so it would be good for you to run ACT and confirm the drive capability before moving on to using asbenchmark (included in the latest tools package).
I am not running on a real SSD. The benchmark is running on a NVMe-OF device backed by DRAM on the target side (The bandwidth is limited by RDMA over IB link and the latencies are in O(10usec)).
Thanks for the ACT pointer. Will take a look and report back.
I created per-thread pool of open file descriptors for ssd_fd_get() and ssd_fd_put() and now the lock bottleneck is gone. With this change, I can get around 900 KIOPs before occupying all 32 hyperthreads.
Does these number make sense? How many IOPs would you expect to get with this number of HTs on a fast SSD?