Aearospike write/read would be failed through the load balancer

Summary

I’d like to do write/read to aerospike cluster through the load balance. But it does not go well. Could you take an advice?

Conditions

  • aerospike version: aerospike server community edition v3.15.01
  • OS: CentOS 7.4.1708

The aerospike access image is like the below.

remote client (VIP) =>load balancer => aerospike cluster

Symptom

When C or PHP client would access to aerospike cluster through the load balancer, there is no connection error, but getting data or putting data would be failed.

Instead of C or PHP client, using aerospike-tool(like aql or adadm), the access itself is succeeded, but AQL execution would be failed.

When using aerospike-tool from the remote client, the access itself was succeeded like the below. *aerospike_vip means the vip of the load balancer.

[cocoamaemae@remote_client ~]$ aql -h aerospike_vip
Aerospike Query Client
Version 3.15.0.3
C Client Version 4.2.0
Copyright 2012-2017 Aerospike. All rights reserved.
aql>

However, aql was failed like the below.

aql> select * from hoge.hogehoge
Error: (7) AEROSPIKE_ERR_CLUSTER_CHANGE

When the aql query was failed, the below log was confirmed in the aerospike server side.

Nov 10 2017 09:27:41 GMT: WARNING (job): (job_manager.c:513) job with trid 2327940182733486773 already active
Nov 10 2017 09:27:41 GMT: WARNING (scan): (scan.c:622) basic scan job 2327940182733486773 failed to start (4)

The aerospike.conf is like the below. Only shows network.service conf

network {
    service {
        address any
        port 3000
        reuse-address
        access-address ****(node ip address)
        alternate-access-address ****(ip address entering into node )
    }

Why C or PHP client could not get or put data?

Why aerospike-tool could access but aql execution would be failed?

Don’t put a load balancer between aerospike client and aerospike server. Any aerospike client (AQL uses C Client underneath) makes connection to one node in the cluster, finds out IP addresses of all nodes in the cluster and then opens direct connections to each node in the cluster. It then receives the data distribution info from the cluster and knows exactly which node has data and goes directly to it. When you put a load balancer in the middle, the load balancer decides which node to send the connection to. You will trip the entire scheme. Effectively, Aerospike has a built in load-balancer by design!

Absolutely, I considered the server constitution not using a load balancer. But now I’m using PHP and C as aerospike client, and there are only programming samples specifying one node https://www.aerospike.com/docs/client/php https://www.aerospike.com/docs/client/c/usage/connect/index.html In these programmings, I think that PHP or C aerospike client cannot communicate with the aerospike cluster if the specified node was down. Is this recognition wrong? And could you tell me the way to avoid the problem not to connect the aerospike cluster in PHP or C, if the specified node was down.

Consider a 10 node Aerospike cluster … each node (Aerospike process strictly) is running on its own linux server - so has its own IP address:port. So you have 10 IP addresses:port that somehow the Aerospike client, running on a different machine needs to connect to.

In any Aerospike client (C / Java … etc) you make connection to one of the those 10 nodes, that one node replies back with IP address:port of all the other nodes also, then you open connections to rest of them. (Sort of a connection handshake!)

Your question as I understand is - what if that one initial node hardcoded in your client application is down? This is called the seed node. Every Aerospike client allows you to specify a LIST of seed nodes. So you can specify 3 or even all 10. The client will go through that list till it finds one good seed node and then the connection handshake takes over. So even if you have subsequently changed the cluster nodes - added more, lost some, as long as your application finds one good seed node in the list, the client will discover the current cluster and open direct connections to each node.

For C, see this page: C Client API: as_config Struct Reference

So look for the correct connect() call that in its config struct takes a LIST of seed nodes.

Hope that explains it.

Finally I succeeded to implement specifying multiple nodes with PHP and C.

PHP is just like the below.

$node1 = ["addr"=>node1_host, "port"=>3000];
$node2 = ["addr"=>node2_host, "port"=>3000];
$node3 = ["addr"=>node3_host, "port"=>3000];
$conf = ["hosts"=>[$node1, $node2, $node3]];
$ae_instance = new \Aerospike($conf);

C is just like the below.

as_config conf;
as_config_init(&conf);
as_config_add_hosts(&conf, "node1_host,node2_host,node3_host", 3000);

The communication to the cluster got well, so a load balancer was not necessary. Thanks you very much for your help.

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.