The max response time jumps to high when reach 32GB


#1

The max response time jumps to high when reach 32GB


#2

Aerospike 3.3.8 with In-Memory (50GB) + Disk Persistence, on HP G6 server (24 CPU-threads, 72GB memory).

Running 70K TPS pure Insert traffic (1KB record size, Client with 4 threads), and have average response time 0.05ms, max average response time 1~2ms, very stable. That’s great!

It keeps such good situation during the period of growing from 0 to 32GB data. When reach and across 32GB data, the max average response time jumps to 100~180ms, and average response time 0.08ms.


#3

Postby Hanson » Thu Jul 31, 2014 9:57 am

When reach 40GB data, I lower the Insert traffic to 18K TPS (Client with single thread), the max average response time drops to 20~70ms. I then let another thread to Delete the records previous Inserted (several minutes ago) ---- that is 50% Insert + 50% Delete, the max average response time of Insert drops to normal 1~2ms ! It’s very strange.


#4

Postby Hanson » Thu Jul 31, 2014 10:03 am

Typo in my previous posts: “max average response time” should be “max response time”.


#5

Tried again with the latest Aerospike 3.3.12. Got the same behavior when DB size > 32GB:

  1. With pure Insert traffic, the max latency keeps ~100ms, average latency 0.08ms.
  2. With balanced Insert and Delete traffic, the max latency keeps ~2ms, average latency 0.05ms. The statistics interval in above is 3s.

Seems it is related to the memory Allocate (for Insert) and Reuse (Delete and Insert).