The set objects number is not empty but the scan result return 0?

1

The aql result is showing the set has millions of objects, but the scan on the set return no result.

with aql show sets, I found this mismatch happened on only one node and all other nodes is showing 0 n_objects.

aql> show sets
+-----------+------------------+----------------+-------------------+----------------+--------------+----------------------------------------+------------+
| n_objects | disable-eviction | set-enable-xdr | stop-writes-count | n-bytes-memory | ns_name      | set_name                               | set-delete |
+-----------+------------------+----------------+-------------------+----------------+--------------+----------------------------------------+------------+
| 41906480  | "false"          | "use-default"  | 0                 | 0              | "test"       | "test_msg"                             | "false"    |

I am using the scanAll api to scan in the whole cluster. In scan option, I am setting the policy.ConcurrentNodes = true and I can see the job running on that node with n_objects not zero. While the scan is running, no any cluster change or migration happening.

aql> show scans
+----------+-----------+----------------------+----------------------+----------------------------------------+----------+----------+--------+---------------------------+-----------------+---------------+-----------+----------+--------------+-----------------+--------------+-----------------+
| ns       | mem-usage | trid                 | job_id               | set                                    | priority | job-type | module | status                    | scanned_records | job_status    | recs-read | run-time | net-io-bytes | time-since-done | job-progress | job_progress(%) |
+----------+-----------+----------------------+----------------------+----------------------------------------+----------+----------+--------+---------------------------+-----------------+---------------+-----------+----------+--------------+-----------------+--------------+-----------------+
| "test"   | 0         | 18                   | 18                   | "test_msg"                             | 1        | "basic"  | "scan" | "done(ok)"                | 0               | "DONE"        | 0         | 29497    | 30           | 1411115         | 100          | 100             |

I think maybe some operations I did early caused this issue, Below is my operations before the mismatch happened:

  1. At first, I restart a node with cold-start-empty=true, and during the migration lots of writes still running.
  2. During the migration, I did lots of expire with ttl=1 operation on the set test_msg (I was trying to delete all objects in this set by setting the expiration ).
  3. After the migration is done, the objects in the set test_msg is empty in all nodes except the restarted node. So I started a scan again but scan returned little records.
  4. I keep re-scanning the set until no any record, however I can still see the millions of objects by aql show sets