The client cannot work while node connections exceed the hard limit suddenly

Client Libraries C Client Library
1

Hi guys, Aerospike C client suddenly log out “max Node *** connection would be exceeed :300”.I wonder what the problem really is. It seems to me the connections which is binding to server ip /port to should be a small number and all the client commands are sent to server through socket fds of very limited sum.I read someone met this problem on the forum,but I have not figured out how to I fix the problem and how to know how many nodes I am using. Thank you in advance.

Steve

2

The C client limits the number of connections (sockets) to each server. This is configurable in as_config max_conns_per_node (sync) and async_max_conns_per_node (async). Default is 300 connections per node.

For sync commands, the number of connections used is the number of application threads that are running single record database commands plus the threads from the sync thread pool (used in batch/scan/query). Sync thread pool size is configured by as_config.thread_pool_size (default 16).

For async commands, the number of connections used is the number of database commands run concurrently in your event loops.

Use the number of application threads, thread pool size, and concurrent async commands to modify the number of connections your application employs.

3

Hi Brian,

Thanks for your explanation, I still have further questions. My application has only sixty threads that are running database commands including query,scan,read, write and batch operations.The cause could be that scan or batch threads each uses more than one connections ,I suppose. It seems to me very queer that the connections could exceed three hundred threshold.How can resolve the problem and how can I know how many connections are used in client while it works sound.

Best Regards,

Steve

4

Are you running sync commands only ?

Are you using the default as_config.thread_pool_size?

If yes to both, then the maximum connections per node you could be using is 76 (60 + 16).

-Brian

5

Hi，Brain

Thanks for your prompt reply.Yes，I use sync commands only and as_config.thread_pool_size is set 16 as default.I have 3 nodes in cluste. so the reason could be the sum of scan/query/batch numbers * 3 puls other node thread number exceed 76. Is it the same reason why it logs out “exceed 300 connections”?I used many scan or query or batch operations,the actural connections could be greater than 76.If I increase thread_pool_size to a greater value could I reslov the problem. Is it the right way of fixing it?

Steve

6

Which version of the C client are you using?

If the sync thread pool was variable sized, extra scan/batch/query threads could multiply by the number of nodes in the cluster. The sync thread pool is really fixed size, so it blocks scan/batch/query threads until a thread slot becomes available. Therefore, the thread limit should really be 76.

Increasing thread_pool_size will not solve your problem. It sounds like connections are being closed, but the connection counter is not being decremented. Please try with the latest version of the C client.

7

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.