I am trying to run a local instance of aerospike server on a 2 x E5-2630 v3 (32 HT) machine. The attached nvme-of device (/dev/nvem0n1
) is capable of over 2M IOPs for 1K IOs reads – verified by FIO (I am using B:100
in my benchmarks, only reads).
service {
paxos-single-replica-limit 1
work-directory run/work
pidfile run/asd.pid
service-threads 16
transaction-queues 8
transaction-threads-per-queue 8
proto-fd-max 100000
}
namespace test {
replication-factor 2
memory-size 1G
default-ttl 30d # 30 days, use 0 to never expire/evict.
storage-engine device {
# Use one or more lines like those below with actual device paths.
device /dev/nvme0n1
# The 2 lines below optimize for SSD.
scheduler-mode noop
write-block-size 4K
data-in-memory false
}
}
I tried running the benchmark both locally and on a different node. In both cases, I get less than 1M TPS (local: ~700K, remote: ~600K).
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6441 root 20 0 4285896 290072 6272 S 2051 0.4 413:54.88 asd
6981 root 20 0 4853380 9612 4260 S 1088 0.0 15:29.61 benchmarks
And this is how perf top -p asdpid
looks like.
6.82% [kernel] [k] queued_spin_lock_slowpath
1.60% [kernel] [k] _raw_spin_lock_irqsave
1.49% [kernel] [k] tcp_ack
1.47% [kernel] [k] __fget
1.23% libc-2.21.so [.] __memcmp_sse4_1
1.18% [kernel] [k] _raw_spin_lock
1.17% libpthread-2.21.so [.] pthread_mutex_lock
1.07% [kernel] [k] try_to_wake_up
1.01% asd [.] as_index_get_v
Edited for clarification: I am not running on a real SSD. The benchmark is running on a NVMe-OF device backed by DRAM on the target side (The bandwidth is limited by RDMA over IB link and the latencies are in O(10usec)).