Changing Replication Factor

Knowledge Base Operating
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Changing Replication Factor

What is Replication Factor?

In Aerospike, the replication-factor configuration parameter controls the “Number of identical copies of each record (including the master copy) maintained in the entire cluster.”

Typical configurations have a replication-factor of 2 which stores two copies of the data, and ensures immediate consistency and no loss of data availability in the event of a single node failure. Additionally, for example with replication-factor 2 if a node goes down, then some amount of the data will no longer be at the full replication factor. The Aerospike cluster will respond by automatically transferring the data between the nodes to re-establish a full replication factor for all data. The motion of this data is referred to as “migrations”.

Although rare, data unavailability will occur if both the master and replica nodes containing the data fail. Replication does increase the total cost of a cluster. With a replication factor 2, the cluster requires twice the storage capacity of a cluster with a replication factor of 1. Replication also suffers a performance penalty, primarily due to the extra network latency incurred by synchronous replication.

A replication factor of “1″ means that the data is not replicated.

Note:

Refer to the Data Unavailability Formula article for details on the amount of data that will be unavailable when replication factor or more nodes go down.

What are the Key Advantages to running with a higher replication factor value?

Aerospike will keep the number of copies of the record data equal to the value used for replication-factor distributed on multiple nodes ensuring high availability of all data in the cluster. This data persistence is beneficial when experiencing node failures. For example, with a replication-factor of 3, Aerospike keeps three copies of the data. This will allow full data availability if the cluster experiences a failure on two nodes simultaneously (or a node failure while migrations triggered by a previous node failure are still in progress).

What are the Key Disadvantages of running with a higher replication factor value?

Capacity needs will increase when increasing the replication factor value. Which increases the total cost of a cluster. For example, with a replication-factor of 3, the Aerospike cluster requires three times the storage capacity compared with a cluster with a replication-factor of 1. So changing from a replication-factor of 2 to 3 will have an storage increase of 50%, the IO incurred by write transactions will also be increased by 50%.

Performance also suffers when increasing the replication-factor value.
It is suggested to benchmark the performance drop before attempting this change on a production cluster.

The penalty is primarily due to the additional intra-cluster communication incurred by synchronous replication. The fabric traffic will increase as well as the SSD write load. There will definitely be a performance hit when changing replication-factor to a higher value, as that number of replicas will have to be written. This would only be for writes and not reads.

Overall write performance will decrease as there are additional data writes to each additional node. Ideally, the writes to the replicas will be done in parallel but they may come back at different times so it will be slower as it is waiting for each node to reply.

How to configure the replication factor?

The replication-factor is configurable on a per-namespace basis.

The replication-factor should be the same value across a cluster for a particular namespace.

The following example shows how in the aerospike.conf file to set a replication factor of 3 :


namespace <namespace-name> {
    ...
    replication-factor 3
    ...
}

WARNING:

Changes to the replication-factor requires a full cluster restart. A mismatch in replication-factor between nodes can cause migrations to get stuck for a particular namespace.

What is the range of values allowed for the replication factor?

The minimum replication factor is 1 (no replication). The upper limit is the number of nodes (a copy of the data in each node).

How can I configure replication-factor for a namespace to ensure all nodes in a cluster have a replica copy?

There is no direct configuration replication-factor all to specify all nodes in the cluster. In the extreme case where you want to ensure all nodes in a cluster have a replica copy set replication-factor to 128. The value of 128 is assuming that your cluster is smaller than 128 nodes, as the present cluster size is limited to 128.

Can the replication factor change be effected dynamically or is it static?

The replication-factor configuration change is classified as static and unanimous, meaning it can only be set when re-starting a node and has to be set to the same value across all nodes in the cluster.

The best approach to effect a change of the replication-factor on an existing cluster is to take a cluster downtime.

  1. You will have to stop the whole cluster.
  2. Change the replication factor on all nodes in the config file.
  3. Start the nodes one by one with at least a 5 seconds delay between nodes to reduce risk of smaller group of clusters initially forming.

NOTE: It is always recommended to back up the data using the asbackup tool on the cluster prior to maintenance.

What Is the expected downtime?

There will be a required downtime for the whole cluster when changing the replication factor on a cluster.

So the time taken will be the total time it takes for all the nodes to come up.

If the cluster has to cold start then the expectation would be the time required for a cold restart. Please review the following document in respect to cold start: http://www.aerospike.com/docs/operations/manage/aerospike/cold_start

How to change the replication factor with existing data, using a clean empty cluster?

When using a clean empty cluster there is a method using asbackup/asrestore to change the replication factor.

  1. Using asbackup, a backup can be saved of the cluster with original replication-factor.
  2. Using asrestore, restore on a clean empty cluster with the an updated replication-factor setting.

The cluster must be clean or there is potential for old records to be resurrected. Please review the following document in respect to Durable Deletes: http://www.aerospike.com/docs/guide/durable_deletes.html

Will the difference be fixed where the configuration parameter name is ‘replication-factor’ and the namespace info statistic name is ‘repl-factor’ ?

It is anticipated in the upcoming version of 3.15.1 that the discrepancy will be resolved. The get-config command name ‘repl-factor’ will be updated to ‘replication-factor’ and display the actual configured replication factor value. The namespace info statistic ‘repl-factor’ will be updated to read ‘effective_replication_factor’ which may be less that the configured replication-factor as it is the replication factor in effect.

Reference:

Keywords

Replication Factor Durability Namespace Configuration asbackup asrestore

Timestamp

10/30/17