Number of client app threads keeps growing!


#1

by Akram » Thu Jul 24, 2014 6:41 am

Hi,

We use Aerospike C# client API and since we started to use it, I noticed that our application’s (.net app) threads number duplicated many times and it keep growing (we usually utilize around 250 thread but now we are utilizing around 1,700 thread!!).

Is this an expected behaviour? or there is something wrong? and what is the healthy expected growth in the number of threads? please advice?

Please note that we just use below APIs from Aerospike client API:

  • Batch Get
  • Get
  • Put
  • Add
  • Operate
  • GetHeader
  • Exists
  • Delete

Regards,


#2

Out of the commands listed, AerospikeClient only spawns threads for “Batch Get” (to read records from multiple nodes in parallel). These threads are retrieved from the standard C# ThreadPool (ThreadPool.QueueUserWorkItem()). When ThreadPool finishes running the task, the thread is placed back into the pool. You can limit ThreadPool size via ThreadPool.SetMaxThreads().

Also, make sure AerospikeClient is only instantiated once and this instance should be shared across all threads that you application generates.


#3

by Akram » Sun Jul 27, 2014 3:54 am

AerospikeClient is already instantiated once for all threads (singlton).

could the batch get cause that jump in number of threads ? and is 1700 thread is healthy number?!! I think that there is something wrong

please also note that our application is clustered at 8 app servers and each server is sending around 1,500 read rps and around 60 write rps.


#4

Keeping 1700 cached threads does seem excessive, but AerospikeClient does not control this. The C# runtime library class ThreadPool determines how many threads remain cached. You can control the max threads with the following command at your application startup.

int workerThreads; int portThreads; ThreadPool.GetMaxThreads(out workerThreads, out portThreads); ThreadPool.SetMaxThreads(, portThreads);

Perhaps your platform or .NET framework ThreadPool version is not reusing threads properly?

Are you observing performance or memory consumption problems when the thread count is high?

How many concurrent threads (generated by your application) are running database commands?

Is your application using ThreadPool directly for some other reason?

You may want to replace “batch get” with multiple record get calls and observe the results. There are cases where single records gets will outperform a single “batch get” (especially with small batch requests).


#5

Hi @iamonthenet

Can you please throw some more light on the case where single record gets will outperform single batch get.

Regards, Dragon


#6

Batch get maps keys to server nodes and sends requests to those nodes in parallel threads spawned from the transaction thread. The responses must be handled and assembled in a thread safe manner. This does add some overhead.

Single record gets can run in the same transaction thread because the request is sent to a single node.

Therefore, small batches and thread resource limited systems or limited thread pools can perform faster using single record gets.

Each system is different, so it pays to benchmark the performance differences.