Hi Team - I have 4 node cluster and in client side below error is frequently coming. Please help to reslove this
com.aerospike.client.AerospikeException$InvalidNode: Error -3: Node not found for partition
Could you share your server and client versions?
The java client had an issue that could erroneously generate these errors prior to 4.3.0. See CLIENT-1053 in the client release notes.
I am seeing the same error in
Java Client 5.0.0 Server: community-5.6.0.4-el8
/Library/Java/JavaVirtualMachines/jdk-11.0.1.jdk/Contents/Home/bin/java “-javaagent:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar=54851:/Applications/IntelliJ IDEA CE.app/Contents/bin” -Dfile.encoding=UTF-8 -classpath “/Users/nareshmaharaj/Documents/aerospike/projects/hello world/target/classes:/Users/nareshmaharaj/.m2/repository/com/aerospike/aerospike-client/5.0.0/aerospike-client-5.0.0.jar:/Users/nareshmaharaj/.m2/repository/org/gnu/gnu-crypto/2.0.1/gnu-crypto-2.0.1.jar:/Users/nareshmaharaj/.m2/repository/org/luaj/luaj-jse/3.0/luaj-jse-3.0.jar:/Users/nareshmaharaj/.m2/repository/org/mindrot/jbcrypt/0.4/jbcrypt-0.4.jar” OperateOnMap
Working with record key:insurance:t2:1:897f41c770078cf71f884907893828a73625710a
Working with record key:insurance:t2:2:10eb8d4c173292ff68fdab697d60d968423aba94
Working with record key:insurance:t2:3:f38d00a6f3be37bdcf958a74173fd933cf1e0168
Working with record key:insurance:t2:4:3bf62f59f82692309578e2793744d6691c995b73
Working with record key:insurance:t2:5:a5ca7823e709a8526e4742ca21dfc94f8fc30902 Exception in thread “main” com.aerospike.client.AerospikeException$InvalidNode: Error -3,1,30000,0,0: Node not found for partition insurance:2725 at com.aerospike.client.cluster.Partition.getSequenceNode(Partition.java:191) at com.aerospike.client.cluster.Partition.getNodeWrite(Partition.java:157) at com.aerospike.client.command.OperateCommand.getNode(OperateCommand.java:42) at com.aerospike.client.command.SyncCommand.executeCommand(SyncCommand.java:76) at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:64) at com.aerospike.client.AerospikeClient.operate(AerospikeClient.java:1305) at OperateOnMap.main(OperateOnMap.java:87)
Process finished with exit code 1
Hits the issue when namespace.set PK is 5 Works fine in aql so suspect client driver
Code as below:
import com.aerospike.client.*;
import com.aerospike.client.policy.ClientPolicy;
import java.text.ParseException;
public class ErrorOnPartition {
public static void main(String args []) throws ParseException {
String host1 = "..amazonaws.com";
String host2 = "..amazonaws.com";
String host3 = "..amazonaws.com";
int port = 3000;
Host[] hosts = new Host[] {
new Host(host1, port), new Host(host2, port), new Host(host3, port)
};
AerospikeClient client = new AerospikeClient(new ClientPolicy(), hosts);
for (int i = 5; i <= 5; i++) {
Key key = new Key("insurance", "t2", i);
System.out.println("\nWorking with record key:" + key);
// PNR Full name
Bin name = new Bin("name", "Mr F Apple");
// Flight Info
String binName = "flight";
client.put(null, key, name);
}
}
}
AerospikeClient finds a valid seed node and then asks that node to provide its other peers in the cluster. The other seed node hostnames are not used unless the first seed node fails.
The peer IP addresses are sometimes AWS internal IP addresses which are not accessible from outside AWS. If so, the client’s view of the cluster will not be fully formed and errors like these will occur.
To verify this is the case, add this line to your code example before creating AerospikeClient.
Log.setCallbackStandard();
This will print cluster tend log messages to stdout.
Nice:
2021-05-28 06:52:36 BST INFO Invalid address 172.31.17.250:3000. access-address is probably not configured on server.
2021-05-28 06:52:37 BST WARN Add node 172.31.37.16 3000 failed: Error -8: java.net.SocketTimeoutException: connect timed out
2021-05-28 06:52:38 BST WARN Add node 172.31.40.221 3000 failed: Error -8: java.net.SocketTimeoutException: connect timed out
2021-05-28 06:52:38 BST INFO Add node BB98E1515796F0A 15.237.63.37 3000
2021-05-28 06:52:39 BST WARN Add node 172.31.37.16 3000 failed: Error -8: java.net.SocketTimeoutException: connect timed out
2021-05-28 06:52:40 BST WARN Add node 172.31.40.221 3000 failed: Error -8: java.net.SocketTimeoutException: connect timed out
Moved the app to the AWS node and all worked well. Thank you very much
If we have a single host h1 which has public address p1 and internal address n1 we can write data to all partitions using p1 and n1 as all partitions exist on h1.
As soon as we add new hosts h2 and h3 and configure the replication / node communication on internal addresses n1,n2,n3 for all hosts h1,h2,h3 the partitions on h1 now get mapped to include additional hosts h2 and h3.
The app can now only publish data on the internal addresses n1,n2,n3 as using p1,p2,p3 outside of the VPC the client app will complain about connection issues to the private addresses. So it appears the partitions are mapped to the host on the private addresses n1,n2,n3 which makes sense as this is what is in each config file for each node.
Ideally how do we set the system up so that writing data can be from an app inside the VPC using n1,n2,n3 or outside using the public address for e.g. p1,p2,p3?
Thank you for taking the time to explain.
Re: https://docs.aerospike.com/docs/operations/configure/network/general/
If I change the private and public address to:
service {
address any
access-address x.x.x.x
alternate-access-address y.y.y.y
}
where it suggests y.y.y.y is public , the public facing app will not connect.
if I use:
service {
address any
access-address y.y.y.y
alternate-access-address x.x.x.x
}
both private facing and public facing apps can send data.
Is there a misunderstanding here?
Kind regards
Naresh
You would need to instruct your client app to use the alternate-access-address through the use-alternate-services config (or something similar).
By default, nodes will broadcast the access-address, and only if client requests the alternate one (through use-alternate-services) would the alternate-access-address be returned. The client app of course need to have a network route to get to the IP.