I believe the issue is the “FAIL_ON_CLUSTER_CHANGE” policy for scans and/or secondary index queries.
See KB for details.
The current implementation of scans and queries cannot ensure all available data matching the request will returned, nor can it ensure that the returned data is only returned once while migrations are ongoing. So the ‘FAIL_ON_CLUSTER_CHANGE’ policy prevents these anomalies by not allowing these APIs. You can change the policy so that it does best effort during these events.
Replacement server APIs are currently being planned for scans and queries that will be capable of ensuring non-duplicated delivery of all available data during disruptions/migrations.