Multiple aerospike_connect() from 1 host leads to uncontrolled too many connections

Hi,

Configuration -

  • 1 aerospike instance per node (4 nodes)
  • 1 host makes below per aerospike node -
  1. as_config_init
  2. as_config_add_host
  3. aerospike_new
  4. aerospike_connect() calls per node
  5. Single event loop is shared among 8 threads. event loop capacity is set to 8.
  6. aerospike_key_operate_async() with pipeline enabled is called on with same event loop.

Observation -

  • Since 64 connections by default with pipeline enabled, with 4 nodes - seeing 1040 connections max. Most of the connections are unused, some connections have netstat send-q piled up.

Expected behavior -

  • Connections as per client’s documentation code, it should be 64x4 = 256 total.?

Are we using the C client correctly where we are instantiating as_cluster per node from 1 host? I noticed that aerospike C client has some global variables.

Does aerospike C Client discover a seed automatically and add it to the cluster? I see “Add seed xxx” in my logs. All 4 nodes have their own instances, how to avoid see discovering?

Yes, all client discover the cluster automatically from the seed node list.

If you are new to Aerospike, please watch this 20 min bootstrap training video: 02 - Intro To Aerospike - YouTube.

Thanks @kporter for the video. It doesn’t touch upon what I am looking for. I have 2 single node cluster, when c client connects to 1 node, it is also adding a seed for node2 that I don’t want it to do. Anyway I can avoid this? The reason is - we use C Client to do put/get on particular node. No replication factor is involved, no failover is involved.

If a record is sent to a specific node in the cluster and that node does not own the record, the record will be forwarded to the node that does own the record.

The only way to guarantee a record stays on a node is to use a single node cluster. In your case, create 2 separate clusters where each cluster contains a single node.

I have 2 separate single node clusters. However, aerospike_connect() to node1 is adding a seed node2 to its list. How does it discover other nodes and add them as seed nodes even in a 2 independent single node cluster scenario? Please check my original description and see if I am missing anything or doing it incorrectly.

I think your two nodes have formed a cluster, the server node is informing the client of other nodes in the cluster.

You can test by running:

asadm -h <ip of server node> -e info

Look at your server logs. This log message will show your cluster size.

Mar 25 2020 00:59:51 GMT: INFO (info): (ticker.c:168) NODE-ID bb96f695d290c00 CLUSTER-SIZE 2

If your cluster size is more than 1, then you have joined your nodes into the same cluster.

oh no! found this in the log - INFO (info): (ticker.c:168) NODE-ID bb9ba91cd3e1600 CLUSTER-SIZE 2

how do I make them 2 independent single node clusters? 2 nodes are running as 2 separate clusters of size 1, atleast intended to.

2020 01:02:07 GMT: INFO (info): (ticker.c:168) NODE-ID bb9ba91cd3e1600 CLUSTER-SIZE 2

This is what I have in aerospike.conf - I didn’t change anything related to heartbeat, its default.

network { service { address any port 3000 }

heartbeat {
    mode multicast
    multicast-group 239.1.99.222
    port 9918

    # To use unicast-mesh heartbeats, remove the 3 lines above, and see
    # aerospike_mesh.conf for alternative.

    interval 150
    timeout 10
}

fabric {
    port 3001
}

info {
    port 3003
}

}

Nodes are joined by multicast-group. Use different multicast-group addresses for each of your nodes to create single node clusters.

Aah! Thank you so much.

Making the multi-cast group address unique per node fixed it to be 2 single node clusters. This also fixed the mystery of ‘?’ number of aerospike connections.

Actually, you should switch to a mesh configuration and not set any seed nodes.

https://www.aerospike.com/docs/operations/configure/network/heartbeat/#mesh-unicast-heartbeat

Also you can set a different cluster-name for each node to further protect them.

https://www.aerospike.com/docs/reference/configuration/cluster-name