High IOWait and maxing out at 5k writes per second


Okay so now it seems that our hosting company gave us an stable environment, but we still can’t get the performance above 5k Writes per second.

We got 3 frontend servers, which processes logfiles and writes to Aerospike. These 3 servers are not maxed out on performance, and also the aerospike servers doesn’t looks to be maxed out on all other parameters but IOWait, which are at ~75% ?

So I was looking for what IOWait actually means and how it is related to Aerospike? Could our performance issue be hiding here?

# Aerospike database configuration file.

service {
        user root
        group root
        paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
        pidfile /var/run/aerospike/asd.pid
        service-threads 4
        transaction-queues 4
        transaction-threads-per-queue 4
        proto-fd-max 15000

logging {
        # Log file must be an absolute path.
        file /var/log/aerospike/aerospike.log {
                context any info

network {
        service {
                address any
                port 3000

        heartbeat {
                mode multicast
                port 9918

                # To use unicast-mesh heartbeats, remove the 3 lines above, and see
                # aerospike_mesh.conf for alternative.

                interval 150
                timeout 10

        fabric {
                port 3001

        info {
                port 3003

namespace audience {
        replication-factor 2
        memory-size 56G
        default-ttl 0 # 30 days, use 0 to never expire/evict.

#       storage-engine memory

        # To use file storage backing, comment out the line above and use the
        # following lines instead.
        storage-engine device {
                device /dev/sdb
#               filesize 16G
#               data-in-memory true # Store data in memory in addition to file.


Can any give me a hint where to look or try?


High IO wait could indicate that the disk isn’t able to handle what you are pushing to it. Have you run ACT to determine what kind of performance you could expect from these disks?