Aerospike Benchmark POC

Benchmarks Aerospike Server Benchmarks
1

Continuing the discussion from How to test and benchmark Aerospike (vs. Couchbase):

Hi, This post is related to what am doing, I have setup Aerospike and configured c and Java clients, but am not sure how to run benchmark tests against the DBs. First I didn’t understand what are the two different steps - load data and stress / performance test against the namespace using the AS clients.

How to get to know what is the exact performance being delivered - because AMC dashboard has Total TPS and successful TPS, which one should I consider,

Otherwise simply to try I ran the command shown in the benchmark example page, which ran for 15 mins and stopped with the below error:

“2015-12-07 16:18:37 ERROR Server is currently in readonly mode. Shutting down...” I couldn’t understand why? all of a sudden?

Same time I wanted to use NVMe SSD, which it was not allowing, when i insert a new namespace with the ssd device settings provided as below:

storage-engine device {
           device /dev/nvme0n1
            write-block-size 128K
            scheduler-mode noop
        }

DB doesn’t start properly, the cake message doesn’t appear in log file.

I badly need some help.

thanks

2

When you get this message, could you run adadm -e "info" and provide the output?

3

Before running a benchmark, data should be loaded into the database so that it can be read back during the load generation of the benchmark.

The following command will load 10 Million Objects into the database. Each object is 1400 bytes. The run_benchmark tool will use 8 threads to insert the data.

./run_benchmarks -h 127.0.0.1 -p 3000 -n test -k 10000000 -b 1 -o B:1400 -w I  -z 8

Running the load. The following command will run 80% reads and 20% writes against the database loaded with the previous command.

./run_benchmarks -h 127.0.0.1 -p 3000 -n test -k 10000000 -b 1 -o B:1400 -w RU,80  -z 8

There are different ways to get performance data from the tools and the database.

The AMC dashboard provides totals and the number of reads and writes per second. If you want more details on performance there are other tools that can be used as well.

The asadm tool can display some summaries of reads and write performance and latencies per node using the “show latency”. http://www.aerospike.com/docs/tools/asadm/

asadm -e "show latency"

The asloglatency tool can be used to provide performance histograms from the Aerospike log file on each node of a cluster. Below is an example use of the tool. http://www.aerospike.com/docs/tools/asloglatency/

 For write performance

asloglatency –t 10 –n 7 –e 1 –h writes_master  

For read performance

asloglatency –t 10 –n 7 –e 1 –h  reads

The run_benchmark also has an option to display a histogram data with the -latency option. These results will be from the client perspective.

The error you are experiencing may occur because the database has reached a stop write limit.

“2015-12-07 16:18:37 ERROR Server is currently in readonly mode. Shutting down…” I couldn’t understand why?

Use

asadm –e “info”

to provide more details.

Storage configuration:

The storage-engine configuration looks fine. The aersopike.log may provide more details of the problem. It is also good to zero out your drive before using the drive.

# For devices that do not support TRIM:
dd of=/dev/zero if=/dev/nvme0n1 bs=1M

# For TRIM supported devices:
blkdiscard /dev/nvme0n1
4

I shall do the things recommended and get back to you with more information from asadm

5

Hi,

I tried to load data using the run_benchmarks as suggested, for testing purpose initially I have selected file based which is in /opt/aerospike/data/ I used the below command:

./run_benchmarks -h 127.0.0.1 -p 3000 -n asdb -k 6000000 -b 1 -o B:1400 -w I -z 8

the tool writes / inserts data to some extent, after which it doesnt complete the job, it kept on looping, at 2015-12-09 08:29:07.649 write(count=5483105 tps=0 timeouts=0 errors=0)

the same message comes after every second, when I checked the size of the data file, that is also not growing, it remained constant at 7.9GB,

my conf file content for the specific db are:

namespace asdb { replication-factor 1 memory-size 128G default-ttl 30d # 30 days, use 0 to never expire/evict.

    #storage-engine memory
    storage-engine device {
        file /opt/aerospike/data/asdb.dat
        filesize 60G
    }

}

I couldnt understand the issue, is this a bug ?

thanks

6

Hi,

I tried loading the data second and it was successful, still couldn’t understand why the tool hangs and repeats on the last step in case of any error?

Also I could see run_benchmarks in java client only, is this not available with C client? One other thing which I wanted to check if there is any pre-built / compiled Java client available for download which doesn’t need Maven build / compile?

thanks

7

There is a similar benchmark utility distributed with the C client as well. https://github.com/aerospike/aerospike-client-c/tree/master/benchmarks

8

Hi,

I shall take a look and try the c tool, otherwise currently am doing benchmark using Java client, I have done couple of tests, 1 AS namespace with NVMe configured as storage option with 1 x NVMe 400GB drive 1 AS namespace with HDD configures and file as storage option, I have loaded around ~50GB of data, followed be run against the data, the performance TPS delivered both storage options is same, I couldnt see any benefit of NVMe for Aerospike, not sure if am doing anything wrong,

storage configuration for NVMe looks as below:

storage-engine device { device /dev/nvme0n1 write-block-size 128K scheduler-mode noop }

not sure whats going wrong, are there tunings that have to be done to get the benefit of NVMe for Aerospike ?

thanks