We have a situation where we recently upgraded a 3 node cluster from 3.4.1 to 3.5.9 and since the upgrade, secondary index queries sometimes fail to find records that we know are available by the primary index.
stat index <ns> <indexname> command shows that the
sync_state=synced, so everything seems to be fine, but querying still comes back empty sometimes.
The rate of query misses is very low(less than 3%), so we don’t think that a whole node has a corrupted secondary index, but we aren’t sure how to recover the missing records.
I am considering a repair index, but I am not sure it would help, and I would like to know the ramifications.
Would a repair fix the problem? Would the node be fully available? How would it respond to queries on the secondary index while it is rebuilding? Would it take as long as a full secondary index rebuild(on that node)?