Upgrading from Citrusleaf client to Aerospike client

Client Libraries
1

I have recently upgraded my application from using the Citrusleaf client to use the Aerospike client, but with that I am seeing a degradation in the performance as compared to the Citrusleaf client. Is their any tuning that I am missing because I have done the migration honoring all the operations and configurations the old client was using ?

2

Can you attach your aerospike.log for us? You aerospike.log should be saved in the default location /var/log/aerospike/aerospike.log.

Also when did you saw these degradation? Was it during the boot up of the Aerospike client?

Jerry

3

Hi, I can’t see this log file. Is it a client side logging or server side logging ? May be because my application does not have the permissions to create the log files. Can the performance impact be due to the log files ? Is there a way to disable this logging ?

Also I am using the aerospike client 3.0.27 on Citrusleaf server 2.6.15-1.

4

Which client are you using? Java?

5

Yes, Java client. Earlier also I was using Citrusleaf java client(2.0.23CP) only.

6

Hi Ankit, -Can you attach your aerospike log file ? It’s on the server side at the path mentioned by Jerry.

  • Also did you change anything in server conf file after up-gradation? Please attach conf file as well from the path “/etc/aerospike/aerospike.conf”

-Anand

7

Hi Ankit, It was nice talking to you over skype. Here are my answers to the questions you have.

  1. If same configuration works with 2.x clients why not with 3.x clients, – It should work with same configuration for either of the client versions
  2. Will the old clients work fine with new server ?
  • Yes old client would work fine with the new server but we recommend you to use new client with new server .
  1. Bench mark which we did and here are the result data, It was done on single namespace .

Benchmarks for the Aerospike old and new java clients. Test Environment Client: able (Centos 6.3, Oracle Java 64 1.7.0_11) Server1: baker (Centos 6.3) Server2: s1 (Centos 6.3) Server3: s6 (Centos 6.3) Configuration

namespace test { replication-factor 1 storage-engine memory } Tests int-: Write 10 byte string key and single integer value using 50% reads / 50% writes with varying thread count. string50-100: Write 30 byte string key and single 50 byte string value using 10% reads / 90% writes with 100 threads. string5000-100: Write 20 byte string key and single 5000 byte value using 40% reads / 60% writes with 100 threads. Conclusions: My initial test environment was a 2 node cluster with a replication factor of 2. This resulted in a negligible new client performance improvement (0 to 3%). I then added a third node and set the replication factor to 1. This made the servers fast enough that the client was the gating factor.

The integer test showed a new client improvement of 10% for 50 threads, 30% for 100 threads and 31% for 125 threads. The new client uses thread local send/receive byte buffers which scales with the number of threads. The old client uses a fixed size buffer pool capped at 50 buffers. When the old client buffer pool is exhausted, the buffers are allocated on the heap which slows the client down.

Performance for the new client tops out at ~385,000 tps with 125 threads. Increasing the number threads past this point does not increase performance.

The small string test show similar new client performance improvements.

The old client beat the new client by 4% in the large string test. The old client estimates the UTF8 send buffer size as “java string length”. The “java string length” estimation can result in send buffer overflows for European and Asian languages. The new client calculates the send buffer size exactly by performing an extra pass on the string. This was done to minimize thread local memory consumption while supporting non-English languages by preventing buffer overflow errors.

8

Here is the benchmark link for Java client, you can try to run this as check the performance for your use case scenario-

http://www.aerospike.com/docs/client/java/benchmarks.html

Hi Ankit, Here is some comparison analysis we did based on your old client and new client code snippets. The new code is doing extra conversions to preserve the “Store” APIs. We have the following suggestions:

  1. The old code passes in CitrusLeafKey. The new code uses “keyWrapper.getKey()” (getKey() source code is not provided). Recommend passing Key class directly instead of “getKey())”. Also, try to avoid using Key constructor that takes in a object, as this is the slowest Key constructor.

  2. Source code for “keyWrapper.getBinNames()” is not provided.

  3. In “_get()”, old code checks “binNames == null”. New code uses “ArrayUtils.isEmpty(binNames)”. Recommend checking “binNames == null” in new code.

  4. In “set()”, old code passes in Map of bins. New code converts map to a list of bins and converts again to array of bins. This is expensive. Recommend convert map to array directly, or better yet pass in array of bins to “set()”.

  5. In “_multiGet()”, old code converts from keys to keysList. New code does the same conversion and then converts again to array of keys. This is expensive. Just convert directly to array, or better yet pass in array of keys to “_multiGet()”.

Also I would like to share the best practices link which talk about some of the things like not to use Bin constructor which takes an object etc. http://www.aerospike.com/docs/client/java/usage/best_practices.html