Distributing reads across proles is on the java client roadmap, but it’s not the panacea you might perceive it to be. Here are some reasons why:
-
Only reads could possibly benefit. Writes must always go to master, even for hot keys.
-
The read-modify-write paradigm can result in more generation errors if prole reads are allowed. If a key is read from a prole while simultaneously being updated by some other thread/client, the generation could change on master before prole. Therefore, the final write could fail more often on the generation check. If the key is read from master, the race condition between master/prole is eliminated.
-
Prole reads require the client to request/process all replica maps for each node. This would increase the amount of data cached by the client by a multiple of the replication factor. Processing time in the cluster tend thread and key partition retrieval is also increased. Not a deal breaker, but something to keep in mind.
-
There will be no benefit if the number of hot keys approaches the number of nodes in the cluster. Suppose a four node cluster with replication factor of 2 is deployed. If there are 4 (or more) products that are very hot, the distribution would look something like this:
hot : master prole
p1 : n1 n2
p2 : n2 n3
p3 : n3 n4
p4 : n4 n1
The distribution of p1 off n1 to n2 is negated by equally hot p4 which adds to n1 load. Nodes are being swapped, but each node’s read load remains the same.
The only time prole reads provide any benefit is when one key is substantially hotter than any other key in the cluster.