CPU analysis of Java client seems to indicate abormally high usage


#1

CPU analysis of Java client seems to indicate abormally high usage

Situation 1

When profiling the CPU usage of the Java client high CPU usage may be noticed in the following area.

com.aerospike.client.cluster.Connection.readFully()

Explanation

This is normal. Most profilers measure CPU usage using time spent in a method. readFully() is a socket call which spends most of its time in a synchronous wait state (i.e. waiting for a response). It is blocking the thread but the CPU is not actually being used much.

As socket reads are OS native calls they are always in the RUNNABLE state while monitoring thread activity. This is because JVM has no way to know if the native call is sleeping or actually doing something. As such, time passed in the RUNNABLE state counts as CPU time.

Situation 2

Thread.sleep () is called in the code leading to apparent high cpu usage in the following.

com.aerospike.client.util.Util.sleep()

Explanation

The background cluster tend thread (which requests partition maps from the cluster) sleeps for 1 second between iterations. This sleep time is counted as CPU usage although the CPU is not doing anything during the sleep. This is nothing to worry about. This can be validated by changing the tend interval in the client policy from the default (1000ms) to 100ms.

ClientPolicy policy = new ClientPolicy();
bc. policy.tendInterval = 100;
bc. AerospikeClient client = new AerospikeClient(policy, host);

The CPU usage for sleep will decrease and the usage for the tend thread will increase accordingly because the client is tending more frequently.