# Client distribution not uniform

**URL:** <https://discuss.aerospike.com/t/client-distribution-not-uniform/4204>\
**Category:** Operations\
**Created:** [May 29, 2017, 2:40am UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204 "2017-05-29T02:40:20Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Foo34](https://avatars.discourse-cdn.com/v4/letter/f/a9a28c/32.png) [@Foo34](https://discuss.aerospike.com/u/Foo34)\
**Post date:** [May 29, 2017, 2:40am UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/1 "2017-05-29T02:40:20Z")

</div>

Hello,

My clients seems to be not well balanced accross my cluster (see AMC screenshot attached). I’m using the Go client. Data seems to be balanced.

I have performance issue (errors, probably because too many read write in progress).

Is there any solution to diagnose what is the issue ? Find the hot keys if this is the problem ? How to improve performances ?

Thx

 ![](https://us1.discourse-cdn.com/flex019/uploads/aerospike/original/1X/19cdac97e1978bdc4edd8293b2afb442119fc37a.png)

---

<div class="post-metadata">

**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\
**Post date:** [May 29, 2017, 2:53am UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/2 "2017-05-29T02:53:47Z")

</div>

Do you have a stateful firewall between any of your clients? There is an bug that 3.12 addresses that causes these symptom in conjunction with a stateful firewall. If it is a hotkey issue, you should see high traffic/latency/queuing on one node versus the others. Can you share the nature of the performance issue/errors you mentioned?

---

<div class="post-metadata">

**Author:** ![Foo34](https://avatars.discourse-cdn.com/v4/letter/f/a9a28c/32.png) [@Foo34](https://discuss.aerospike.com/u/Foo34)\
**Post date:** [May 29, 2017, 3:01am UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/3 "2017-05-29T03:01:39Z")

</div>

I’m on GCP, I do not have anything specific between nodes.

Problems I have: write error spikes. not huge, but often during a peak of r/w in progress on the node which has lot of connections (.39):

I will try an upgrade to 3.12.1. Let me know if you have others ideas.

---

<div class="post-metadata">

**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\
**Post date:** [May 29, 2017, 4:34am UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/4 "2017-05-29T04:34:21Z")

</div>

Not between the nodes, between the cluster and the clients.

---

<div class="post-metadata">

**Author:** ![Foo34](https://avatars.discourse-cdn.com/v4/letter/f/a9a28c/32.png) [@Foo34](https://discuss.aerospike.com/u/Foo34)\
**Post date:** [May 29, 2017, 6:57pm UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/5 "2017-05-29T18:57:23Z")

</div>

Clients and server are on the zone, sharing the same network.

The cluster upgrade is still in progress, but I already have a non uniform balancing for clients. The latency is higher on nodes which have a high number of connections, but the difference is not huge.

If this is a hot key issue, is there anything to do on aerospike side ?

Do you think than 10k clients for a node is huge ? To say it another way, should I resize the cluster to have more nodes, may be more smaller nodes ? Currently nodes are 32 CPU / 64 G of RAM.

Thx.

---

<div class="post-metadata">

**Author:** ![Foo34](https://avatars.discourse-cdn.com/v4/letter/f/a9a28c/32.png) [@Foo34](https://discuss.aerospike.com/u/Foo34)\
**Post date:** [May 31, 2017, 2:24am UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/6 "2017-05-31T02:24:31Z")

</div>

Changing the shape of the cluster seems to fix the issue: 15 small server instead of 7 big, with a global constant footprint. I still have errors, but less.

If you have any hint why the traffic seems to be badly distributed, let me know.

---

<div class="post-metadata">

**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\
**Post date:** [May 31, 2017, 2:32am UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/7 "2017-05-31T02:32:23Z")

</div>

Well i did mention you could check your histograms and your queue depths historically…

---

<div class="post-metadata">

**Author:** ![Foo34](https://avatars.discourse-cdn.com/v4/letter/f/a9a28c/32.png) [@Foo34](https://discuss.aerospike.com/u/Foo34)\
**Post date:** [May 31, 2017, 2:56am UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/8 "2017-05-31T02:56:45Z")

</div>

Latency is normal (\<5% on 1ms for every operation / every nodes).

For queue depth, I’m not sure what you mean. Rw\_in\_progress can be spiky (see attached graph), but errors are still localized on two / three nodes on 15, which can be explained by hot keys.

![](https://us1.discourse-cdn.com/flex019/uploads/aerospike/original/1X/5fa8c636080a96d6532c4b1e4c1931b1f488bbcd.png)

---

<div class="post-metadata">

**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\
**Post date:** [May 31, 2017, 11:28pm UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/9 "2017-05-31T23:28:49Z")

</div>

[http://www.aerospike.com/docs/reference/serverlogmessages/](http://www.aerospike.com/docs/reference/serverlogmessages/) can you post some logs and let us know the time stamp you experienced the issue so we can cross reference the logs

---

<div class="post-metadata">

**Author:** ![Foo34](https://avatars.discourse-cdn.com/v4/letter/f/a9a28c/32.png) [@Foo34](https://discuss.aerospike.com/u/Foo34)\
**Post date:** [June 8, 2017, 1:50pm UTC](https://discuss.aerospike.com/t/client-distribution-not-uniform/4204/10 "2017-06-08T13:50:20Z")

</div>

I do not have any error in server log (or I did not find them). Errors are timeout (1s). Currently it’s mainly in one of 15 servers. Do you know if I can increase the logging level on a specific component to debug ?