We are experiencing issue where one of our nodes get twice as much “master objects” as the others in the cluster.
Distribution of partitions among nodes seems to be fine. At least according to:
asmonitor -e "pmap [namespace]" it seems that all nodes got more less equal number of partitions.
Then it could be that one of the partitions got significantly more objects than the others, but I don’t know how to prove it. Is it possible to print out information about distribution of objects among partitions?
Even if it seems to be quite likely scenario there are two things which concern me:
- replica object seems to be well balanced across nodes, and if one partition would be significantly bigger then I suspect that replica objects on one of the nodes would be also significantly bigger than on the other.
- what would have to happen to the key values to cause such huge disproportion in partitions size?
Do you have any idea how to track down this issue?