How do I synchronize all metadata across clusters?

Knowledge Base Operating
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Metadata synchronization across multiple clusters in Aerospike

Problem Description

I have multiple clusters that run Aerospike. Some of them are setup for XDR replication as well. I see that Aerospike writes some metadata files in smd folder. How do I make sure that all my clusters have their system metadata in-sync?

Explanation

Aerospike writes system metadata files in the smd folder. The System MetaData (smd) directory contains a catalog of data kept in a distributed, persistant fashion. The format for these files is JSON, for readability. The files include:

  • Secondary index definitions
  • Registered UDF modules
  • User credentials hashes and configured roles if security is enabled
  • Truncate related metadata

Files in this directory can be deleted, and will be re-created when the server starts.

Solution

The metadata folder is not synced across different clusters. This includes clusters that are setup for XDR. If you do intend to keep the clusters in sync, you have the following scenarios to consider:

  1. Copy all the files over before the new clusters are started-up.
  2. Execute the same operations on both clusters.
  3. Review the steps below for details on syncing existing clusters:
  • Secondary Index (sindex.smd): You would need to re-create the secondary index definitions. Copying the files over and triggering a rolling restart of the cluster will not work as the sindex smd merge follows a majority consensus, so it would work if the whole cluster was shut down and nodes were brought up with the copied smd sindex definitions one node at a time.
  • UDF (udf.smd): You can sync this file simply by re-registering the module on the cluster that has it missing. There is no requirement for a node restart.
  • Security - ACL (security.smd): You can re-create the users, roles on the other cluster if security is already enabled. If security is not yet enabled, you would need to enable it first before creating the users and roles.
  • Truncate (truncate.smd): You would need to re-issue the truncate command on the required clusters. Note that this file contains timestamp details, thus copying this manually and re-starting might cause unexpected issues.

Keywords

metadata smd sync synchronize

Timestamp

10/27/2017