There is a known issue on 3.14.0 that impacts secondary index:
- Regression - [AER-5698] - (KVS) Records created via migrations are not added to secondary index. Fixed in version 18.104.22.168
If this is the issue you encountered, as a workaround, on version 3.14.0, you could restart the node that has less keys/entries and that should force the sindex to be rebuilt.
Secondary index are built on both master and prole transactions… so nodes would be able to take over server secondary index queries when a node leaves the cluster.
The bug caused records ‘created’ through an incoming migration to not have their secondary index created. So if you added an empty node, most records on that node would be created through incoming migrations and would be missing their secondary index.
Upgrade to version 22.214.171.124 and you should be fine, based on your description and the data you shared I am fairly certain this is the issue you did hit.