Read from all nodes in a cluster, monitoring and implementation issues

Greetings, I’m pretty new to Aerospike and I’ve been playing with it in order to see how good it is. I really like it so far, but I got stuck when trying to test out my cluster. So I currently have a cluster of 3 nodes - all running on containers - which I’m using via my C# API. Rather than explaining my configuration in words I decided to provide these snaps of it. So the problem arises when checking my reads and writes over amc. It seems to show that my reads are being done via all nodes in a cluster while my reads are done only via a single node - which isn’t what I want to achieve. Did anyone have this issue? Pics below show my amc problems.

Aerospike configuration :

Docker ps result:

Note: I’ve mapped these to my local ports so I can access their session on amc over the localhost

Environment: OS: Ubuntu 18.04 Docker version 18.09.6 Aerospike version 4.5.2.2 Dotnet version 2.2.204

AMC :

So as you can see when performing a test write of around 100000 records I’m using all nodes to perform this action, but while reading a single record a 100000 times I’m using only one node. I’ve tried with reading all the keys one by one per say(I’m intentionally doing this rather than reading a RecordSet), but it also didn’t work. Furthermore, I’ve set the client policy like this(maybe I’m missing something here): Screenshot%20from%202019-06-04%2015-53-47

Hopefully someone can notice what a newbie like me is doing wrong here. If anyone needs more info I will happily provide it. Thanks.

Watch this 24 min video: https://www.youtube.com/watch?v=PA7PGWphW8M

Thank you for the speedy reply, what I got that regarded my issue from this video is “when you read a record the client library just goes to the master node”. I’m aware that the data is distributed over the cluster and when getting it all the master has to access the data on those partitions, but ultimately does this mean that the master node is doing the heavy lifting regarding the reads or ? Can this be manipulated in some way through the configuration or policies ? Also it says that it chooses the master based on where the read’s key location is - when reading all the data, these keys should be evenly distributed - shouldn’t that involve other nodes into that read directly ? Thanks.

Every record belongs to a unique partition id - there are 4K partition ids. If you have a 10 node cluster, each node will be a master of 4K/10 partitions. So recA may be on parition 233 which has node1 as its master and will always be read from node1, whereas recB may be on partition 532 which may have node5 as its master and therefore always read from node5. in your test, you were reading the same record again and again so it kept going to the same node which had the record (was the master for that record’s partition). Aerospike is not “Master-Slave” architecture like Hadoop - its a distributed master “shared nothing” architecture. With this in mind, perhaps watch the video one more time… :slight_smile: may clarify things further.

Ah… when reading multiple records - say using batch index read API - client will send requests to the different masters in parallel and they will each reply back with the record or records they have. So yes, if you read different keys, even as sequential individual read requests, they will go to appropriate (different quite likely - based on the digest (hash)) nodes which are the masters for that record. This is what is termed as “single hop to data”.

Thank you this was very insightful. Finally I have one question about write policies - what is the deeper difference between MASTER_PROLES and RANDOM ? Except including the master node prior, they both use all the nodes available ultimately ?

RANDOM: In AP mode, you can set replication factor to number of nodes - e.g. rf=10 in a 10 node cluster. Now each node will have every record either as a master or replica. With read RANDOM, you are asking the client to go to any any node at random, and read the record - master copy or replica copy - you don’t care. This mode is rarely used - but some use it implement a small data set as a fast lookup table. Reading from MASTER is the default. Using MASTER_PROLES - you are asking the client to read from either the Master node or the Replica node, the node that has the copy of the record based on RF value. Typically, RF=2, so you have another node besides the master that has the copy of the record. So you are asking the client to round robin between master and replica - this distributes the read load at the possible expense of reading stale data on replica in AP mode if a read happens to come simultaneously as a write on the master to the same record which is in the process of getting replicated. i.e. as a timeline: write comes on nodeA(master) - read comes on nodeB(replica) - nodeA replicates write to NodeB the write - the read served by nodeB was not most up to date. If the data-model doesn’t care about it, then use it. Gets you faster read performance.

BTW - I assume you are using the Community Edition. When you download CE, you can register and get free access to a longer ( I think it is about 80 min) - “Intro to Aerospike” course on Aerospike Academy. It will be worth it for you to do that.

Okay, thank you for the info, I really appreciate it :slight_smile: