Migration per namespace?

Feature Discussion Migration
1

Hi, I have to 2 namespaces in my database. One is very big with 3M records, and the other one is very small. I am using scans to read the data from the small namespace. On a cluster change event, the migration process takes about 2 minutes, because of the big namespace migration. The scans of the small namespace are critical, and must be precise, so doing them during a cluster change is not a valid option. Theoretically, I would want the migration of the small namespace to complete first, and then to scan it, but from what I see, only after the full migration process completes, the cluster is considered “stable”, and scans can work.

Is there an alternative to waiting for the full migration process to complete just for the small namespace?

2

There is no alternative to prioritize migration for a specific namespace as of Aerospike version 3.7.0.2.

That being said, your request of ability to prioritize a namespace for migration has been considered as a valid feature request. Here is the Jira reference - AER-4680

Also, would like to mention that scan’s migration detection has been enhanced in the the latest server builds (3.7.0.1 and above). This will help scan to work for the namespace that finished migrations.

The other feature in 3.7.0.1 is % level marking of namespace migration status in the server logs.

http://www.aerospike.com/enterprise/download/server/notes.html#3.7.0.1

[AER-3639] - (Migration) Added new migration stats to accurately reflect migration progress.

[AER-4508] - (Scan) Enhance scan’s migration detection.

Additionally, specifying the smaller namespace before the bigger one in Aerospike configuration will have a higher chance of finishing migrations earlier. But, that would require a cluster-wide restart, so let’s just park it as a possible consideration while deploying new clusters.

3

Hi, thanks for the reply.

About what you said here: “This will help scan to work for the namespace that finished migrations”. Let’s say I am scanning a namespace with a flag “fail on cluster change”, and the migration of this specific namespace has finished, but migration for other namespaces hasn’t finished. Will it fail or not? If the answer is yes, than I think this is a very important feature to add.

4

Scan will work for the namespace that has finished migrations starting version 3.7.0.1.

Now, ‘Fail on cluster change’ flag is for situations when the cluster size changes due to a loss of a node or cluster integrity issues and re-trigger migrations for all namespaces and scan will still fail till the namespace migrations are completed.

The feature will be to prioritize the namespace for migration. Scans working during cluster size changes is up to the user’s discretion. Our recommendations are still to keep the flag true as the partitions are influx during migrations, so unless that specific namespace has finished migration, it will not be providing accurate data.

5

Hi, I see you added a new feature: [AER-4869] - (Fabric) Namespace level migration prioritization

Is there a documentation about how this feature is used?

UPDATE: Found it -

http://www.aerospike.com/docs/reference/configuration/#migrate-order