Look what happens if I increase to 100 threads:
AMC shows it alternates between 100% successful writes and 0% success…
There is some kind of fluctuation visible in the latency logs too (between ~3000 and ~1800):
13:23:02 10 0.63 0.04 0.00 3008.7
13:23:12 10 0.31 0.01 0.00 2498.8
13:23:22 10 0.61 0.04 0.00 3327.9
13:23:32 10 0.30 0.02 0.00 1824.5
13:23:42 10 0.56 0.05 0.00 3576.0
13:23:52 10 0.36 0.03 0.00 1878.3
13:24:02 10 0.52 0.03 0.00 3606.9
13:24:12 10 0.42 0.03 0.00 1708.2
13:24:22 10 0.50 0.03 0.00 3645.5
13:24:32 10 0.53 0.06 0.00 1752.1
13:24:42 10 0.35 0.02 0.00 3048.9
13:24:52 10 0.64 0.05 0.00 2607.0
13:25:02 10 0.32 0.01 0.00 2478.3
13:25:12 10 0.66 0.01 0.00 3014.7
13:25:22 10 0.22 0.00 0.00 1748.1
output of top on the server side !!!
Is client running on the same box
- -R
Yes I’m running client and server on the same machine. On windows with VirtualBox running Aerospike.
Both VM and windows report around 60-70% CPU usage
joshfg,
Such spikes shows possibly request backlog and being served in next second … Could you try with client on different box to avoid server contending for CPU with client. Other option would be you taskset to bind both and server and client to different set of of CPU.
$ sudo taskset -p 26599
pid 26599's current affinity mask: ff
$ sudo taskset -p f 26599
pid 26599's current affinity mask: ff
pid 26599's new affinity mask: f
$ sudo taskset -p 26599
pid 26599's current affinity mask: f
If things still do not improve. Can have a run with microbenchmarks enabled
asinfo -v 'set-config:context=service;microbenchmarks=true'
This would dump detailed breakup of where time is spent while executing transaction … in log. You can share few lines of log with that histogram post run.
– R
joshfg,
This is statistics bug. You should look at stat_write_success and err_write_fail* for successful writes and reads respectively in
asinfo -v "statistics"
Will fix this bug.
– R
Hi Raj,
Ok I’ll do that for now!
Thanks for the help on this,
Josh