High IOWait and maxing out at 5k writes per second


#1

Okay so now it seems that our hosting company gave us an stable environment, but we still can’t get the performance above 5k Writes per second.

We got 3 frontend servers, which processes logfiles and writes to Aerospike. These 3 servers are not maxed out on performance, and also the aerospike servers doesn’t looks to be maxed out on all other parameters but IOWait, which are at ~75% ?

So I was looking for what IOWait actually means and how it is related to Aerospike? Could our performance issue be hiding here?

# Aerospike database configuration file.

service {
        user root
        group root
        paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
        pidfile /var/run/aerospike/asd.pid
        service-threads 4
        transaction-queues 4
        transaction-threads-per-queue 4
        proto-fd-max 15000
}

logging {
        # Log file must be an absolute path.
        file /var/log/aerospike/aerospike.log {
                context any info
        }
}

network {
        service {
                address any
                port 3000
        }

        heartbeat {
                mode multicast
                address 239.1.99.222
                port 9918

                # To use unicast-mesh heartbeats, remove the 3 lines above, and see
                # aerospike_mesh.conf for alternative.

                interval 150
                timeout 10
        }

        fabric {
                port 3001
        }

        info {
                port 3003
        }
}

namespace audience {
        replication-factor 2
        memory-size 56G
        default-ttl 0 # 30 days, use 0 to never expire/evict.

#       storage-engine memory

        # To use file storage backing, comment out the line above and use the
        # following lines instead.
        storage-engine device {
                device /dev/sdb
#               filesize 16G
#               data-in-memory true # Store data in memory in addition to file.
        }
}

#2

Can any give me a hint where to look or try?


#3

High IO wait could indicate that the disk isn’t able to handle what you are pushing to it. Have you run ACT to determine what kind of performance you could expect from these disks?