Benchmarking Throughput and Packet Count with iperf3


#1

Benchmarking Throughput and Packet Count With Iperf3

Understanding the performance of a systems resources is important for managing the capacity and operations of Aerospike. This is a quick reference to understand the performance of your network resources. Included in this article is a benchmark setup for both throughput benchmarking and packet count benchmarking. Many benchmarks focus just on throughput. However it is specially important to understand the network packet count limits in a cloud environment like AWS because of the network throttling done by the hypervisors.

Installing Iperf3 on Centos and Ubuntu

You will need to install iperf3 on both of the servers used in the benchmark.

Install Iperf3 on Centos:

sudo yum install iperf3

Install iperf3 on Ubuntu 14.04

sudo apt-get install iperf3

Throughput Benchmark

Server Setup:

sudo iperf3 -s -p 3000

Client Setup:

sudo iperf3 -c 10.0.0.31 -i 1 -t 3600 -V -p 3000  -b 1000000000000


Where 
    -p  port number
    -c server IP address
    -t is time for the test in seconds 
    -i option is the interval for output. 
    -b is the bandwidth in bits per second. This number should be set to the maximum bandwidth for the network interface.

Once the server and client are setup and running, you can use the scripts below to monitor the bandwidth utilization and the packet counts. Many monitoring tools will supply throughput statistics but no packet counts. The scripts below will supply packet count monitoring for the benchmark. When your test has finished running you will want to start the test a second time. However you will want to run the original server machine as a client and the orignal client machine as a server.

Packet Count Benchmark

Server Setup:

sudo iperf3 -s -p 3000

Client Setup:

sudo iperf3 -c 10.0.0.31 -i 1 -t 3600 -V -p 3000  -b 1000000000 -l 100 -M 89

-l option is the buffer size. Actually this doesn't matter too much.   
-M option is the MTU size.   The default is 1500.  Here we set it low to get a high packet count. The throughput is set high.
-p  port number
-c server IP address
-t is time for the test in seconds 
-i option is the interval for output. 
-b is the bandwidth in bits per second.  This number should be set to the maximum bandwidth for the network interface.

Note: The –l option is the buffer size and the -M options are only used in the packet count test.

When your test has finished running you will want to start the test a second time. However you will want to run the original server machine as a client and the orignal client machine as a server.

Sample iperf3 Client output

sudo iperf3 -c 10.0.0.212 -i 1 -t 3600 -V -p 3000  -b 1000000000000
iperf 3.0.7
Linux ip-10-0-0-74 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Time: Mon, 28 Mar 2016 21:54:13 GMT
Connecting to host 10.0.0.212, port 3000
      Cookie: ip-10-0-0-74.1459202053.406100.67077
      TCP MSS: 8949 (default)
[  4] local 10.0.0.74 port 51752 connected to 10.0.0.212 port 3000
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 3600 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   422 MBytes  3.54 Gbits/sec   23    953 KBytes       
[  4]   1.00-2.00   sec  82.8 MBytes   694 Mbits/sec    0   1.10 MBytes       
[  4]   2.00-3.00   sec  82.9 MBytes   695 Mbits/sec    0   1.25 MBytes       
[  4]   3.00-4.00   sec  82.9 MBytes   695 Mbits/sec    0   1.38 MBytes       
[  4]   4.00-5.00   sec  81.8 MBytes   686 Mbits/sec    1   1.21 MBytes       
[  4]   5.00-6.00   sec  83.9 MBytes   704 Mbits/sec    0   1.27 MBytes       
[  4]   6.00-7.00   sec  82.8 MBytes   694 Mbits/sec    0   1.41 MBytes       
[  4]   7.00-8.00   sec  82.9 MBytes   695 Mbits/sec    0   1.48 MBytes       
[  4]   8.00-9.00   sec  82.8 MBytes   694 Mbits/sec    1   1.19 MBytes       
[  4]   9.00-10.00  sec  82.9 MBytes   695 Mbits/sec    0   1.36 MBytes       
[  4]  10.00-11.00  sec  82.8 MBytes   694 Mbits/sec    0   1.45 MBytes       
[  4]  11.00-12.00  sec  82.9 MBytes   695 Mbits/sec    1   1.09 MBytes       
[  4]  12.00-13.00  sec  82.9 MBytes   695 Mbits/sec    0   1.29 MBytes       
[  4]  13.00-14.00  sec  82.8 MBytes   694 Mbits/sec    0   1.42 MBytes       
[  4]  14.00-15.00  sec  82.9 MBytes   695 Mbits/sec    0   1.48 MBytes       
[  4]  15.00-16.00  sec  82.8 MBytes   694 Mbits/sec    1   1.21 MBytes       
[  4]  16.00-17.00  sec  82.9 MBytes   695 Mbits/sec    0   1.37 MBytes       
[  4]  17.00-18.00  sec  82.8 MBytes   694 Mbits/sec    0   1.46 MBytes       
[  4]  18.00-19.00  sec  82.9 MBytes   695 Mbits/sec    1   1.11 MBytes       
[  4]  19.00-20.00  sec  82.9 MBytes   695 Mbits/sec    0   1.31 MBytes       
[  4]  20.00-21.00  sec  82.8 MBytes   694 Mbits/sec    0   1.43 MBytes       
[  4]  21.00-21.78  sec  64.5 MBytes   695 Mbits/sec    0   1.48 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-21.78  sec  2.09 GBytes   825 Mbits/sec   28             sender
[  4]   0.00-21.78  sec  0.00 Bytes  0.00 bits/sec                  receiver
CPU Utilization: local/sender 4.6% (0.9%u/3.7%s), remote/receiver 0.0% (0.0%u/0.0%s)
iperf3: interrupt - the client has terminated

Sample iperf3 Server Output

sudo iperf3 -s -p 3000
-----------------------------------------------------------
Server listening on 3000
-----------------------------------------------------------
Accepted connection from 10.0.0.74, port 51751
[  5] local 10.0.0.212 port 3000 connected to 10.0.0.74 port 51752
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   411 MBytes  3.45 Gbits/sec                  
[  5]   1.00-2.00   sec  82.8 MBytes   695 Mbits/sec                  
[  5]   2.00-3.00   sec  82.8 MBytes   695 Mbits/sec                  
[  5]   3.00-4.00   sec  82.8 MBytes   695 Mbits/sec                  
[  5]   4.00-5.00   sec  82.8 MBytes   695 Mbits/sec                  
[  5]   5.00-6.00   sec  82.8 MBytes   695 Mbits/sec                  
[  5]   6.00-7.00   sec  82.8 MBytes   695 Mbits/sec                  
[  5]   7.00-8.00   sec  82.8 MBytes   695 Mbits/sec                  
[  5]   8.00-9.00   sec  82.8 MBytes   695 Mbits/sec                  
[  5]   9.00-10.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  10.00-11.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  11.00-12.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  12.00-13.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  13.00-14.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  14.00-15.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  15.00-16.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  16.00-17.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  17.00-18.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  18.00-19.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  19.00-20.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  20.00-21.00  sec  82.8 MBytes   695 Mbits/sec                  
[  5]  20.00-21.00  sec  82.8 MBytes   695 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-21.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-21.00  sec  2.09 GBytes   853 Mbits/sec                  receiver
iperf3: the client has terminated
-----------------------------------------------------------
Server listening on 3000
-----------------------------------------------------------

Sample Output from Scripts

Packet Count

./nettp eth0
TX eth0: 9702 pkts/s RX eth0: 4558 pkts/s
TX eth0: 9745 pkts/s RX eth0: 4479 pkts/s
TX eth0: 9738 pkts/s RX eth0: 4493 pkts/s
TX eth0: 9693 pkts/s RX eth0: 4547 pkts/s
TX eth0: 9752 pkts/s RX eth0: 4497 pkts/s
TX eth0: 9745 pkts/s RX eth0: 4492 pkts/s
TX eth0: 9738 pkts/s RX eth0: 4488 pkts/s

Throughput

    ./nettp eth0
    TX eth0: 85845 kB/s RX eth0: 289 kB/s
    TX eth0: 85783 kB/s RX eth0: 289 kB/s
    TX eth0: 85660 kB/s RX eth0: 290 kB/s
    TX eth0: 85369 kB/s RX eth0: 295 kB/s
    TX eth0: 85783 kB/s RX eth0: 288 kB/s
    TX eth0: 85721 kB/s RX eth0: 288 kB/s

Network Packet Count Script

#!/bin/bash

INTERVAL="1"  # update interval in seconds

if [ -z "$1" ]; then
        echo
        echo usage: $0 [network-interface]
        echo
        echo e.g. $0 eth0
        echo
        echo shows packets-per-second
        exit
fi

IF=$1

while true
do
        R1=`cat /sys/class/net/$1/statistics/rx_packets`
        T1=`cat /sys/class/net/$1/statistics/tx_packets`
        sleep $INTERVAL
        R2=`cat /sys/class/net/$1/statistics/rx_packets`
        T2=`cat /sys/class/net/$1/statistics/tx_packets`
        TXPPS=`expr $T2 - $T1`
        RXPPS=`expr $R2 - $R1`
        echo "TX $1: $TXPPS pkts/s RX $1: $RXPPS pkts/s"
 done

Network Bandwidth Script

#!/bin/bash

INTERVAL="1"  # update interval in seconds

if [ -z "$1" ]; then
        echo
        echo usage: $0 [network-interface]
        echo
        echo e.g. $0 eth0
        echo
        exit
fi

IF=$1

while true
do
        R1=`cat /sys/class/net/$1/statistics/rx_bytes`
        T1=`cat /sys/class/net/$1/statistics/tx_bytes`
        sleep $INTERVAL
        R2=`cat /sys/class/net/$1/statistics/rx_bytes`
        T2=`cat /sys/class/net/$1/statistics/tx_bytes`
        TBPS=`expr $T2 - $T1`
        RBPS=`expr $R2 - $R1`
        TKBPS=`expr $TBPS / 1024`
        RKBPS=`expr $RBPS / 1024`
        echo "TX $1: $TKBPS kB/s RX $1: $RKBPS kB/s"
done