Application does not find the replica node

[totola]

we have a cluster with two nodes and replica factor set 2 , that means , every node have the whole data , but when we shutdown one node the application down and do not use another node

we only configure one node as seed (is the shutdown node) , but the same configuration had been used in other cluster ,and everything is just fine ,when we shutdown one node in other cluster , the only difference between these cluster , is replica factor , other cluster replica factor is 1 ,means some partition data cant be reached during the downtime .

in my understanding , i think the client would cache the cluster node list from seed node request , and when one node can’t be reached will turn to other nodes , is there something wrong ?

[neelp]

The node may be unreachable from the client machine. Can you ping the node ip from the client machine? The network/service stanza in the node’s aerospike config file is where you can specify a node’s ip addresses for access from different clients based on where they are on the network . Please check out the doc: https://www.aerospike.com/docs/operations/configure/network/general.

[totola]

the network is fine , cause after the application down , we change the application connection setting , set the alive node as client node , and restart the application ,everything just fine .

we add two nodes in program as hard code , as below

this time we shutdown one node application working fine .

But my question is why client don’t cache all the nodes list ? the hard code setting to add node is not good enough , cause we may add another node in the future . we don’t want to change code or any configure files everytime when we add node to the cluster

[totola]

In other words , does it is necessary, when cluster is only two nodes , we have to configure two nodes as seed nodes to prevent anyone node down .

after some investigation , i guess , in two node cluster , after one node down , the alive node can’t be majority itself , so paxos vote can’t select a new cluster view . isn’t it ?

[meher]

Couple of important points:

• From the client’s perspective, a client does require a list of seed nodes when it starts up. It will use the first seed that it can connect to and will discover the rest of the cluster as such. There is no mechanism to cache previously ‘known’ nodes upon a restart… this could have other undesired effects (reaching the wrong cluster for example, or having nodes that are then deployed across different clusters).

• If a node in the cluster goes down while the client is running, the client will automatically continue against the remaining nodes in the cluster. This is a completely different heuristic compared to ‘restarting a client’. One common mistake is to not configured the access-address on cluster nodes running on public clouds like AWS. A Client connecting from a different VPC and not having access to the private IP addresses will only be able to properly connect to the cluster. If seeding the Client with a public IP address, the Client will only be able to connect to that node and will proxy all the transaction through that same node, but will not be able to connect to the other nodes.

Feel free to share your database configuration or describe the setup… the Client should also provide log messages (make sure you subscribe to the log facility) so that you can see why the Client stops connecting to the cluster after one node goes down.

[totola]

i could understand the first points ,that’s ok , but in our case does not match the second point , our nodes and application are all in vpc and communicate through private network , do not use public ip address , in our cases , when a node in the cluster goes down , the client is running , but do not automatically continue against the remainning nodes , that’s why i come here for help .

[meher]

Understood… if all are in the same VPC, then something else may be going on. Are you able to run the Java benchmark tool successfully and observe what happens when a node goes down? It should certainly continue against the remaining node (while throwing some errors because it cannot ‘tend’ to the node that went down but would direct all transactions to the remaining node).

If this is confirmed, then it would seem the issue is on the way the application may be continuously re-instantiating the Client object, for which you would need at least 1 IP address of a node that is up.

Some users inject such IP ‘externally’ or use a round robin DNS entry which will then always return the IP address of a node that is up behind it.

Hope this helps.

[totola]

the application is a task running in aws spark service , i guess the reason is one of below

1 every thread in task try to connect the aerospike everytime, and do not use connection pool , so there is no long connection , for a new client , the seeds node down , it will not know other alive node

2 the application code do not cache the error when one node down , then the error throw to spark ,then spark think the task going wrong ,and recreate a task ,so long connection killed by spark , new task can’t access the seeds node (down node) , it will not know other alive node either

And from the developer’s reply , they sure do not catch the error .

thanks for your help

[meher]

Yeah, that would explain it… using a round robin DNS entry to seed the client should help, if that is possible.

[meher]

Am curious, though, are you using some Spark integration with Aerospike? Or your developers have written custom code to have Spark read/write into Aerospike?

[totola]

yes , they have written custom code to have spark read/write into aerospike

[meher]

Thanks for confirming.