Upgrading from versions prior to 3.13 to versions 4.2 and above while using custom node IDs

Upgrading from versions prior to 3.13 to versions 4.2 and above while using custom node IDs

Problem Description

When using custom node IDs, please note that the feature is removed from 3.13 after protocol upgrade and a new way to set node-id is added in version 3.16.0.1.

If you upgrade to 3.13, and change protocol version, while using custom node IDs (self-node-id configuration parameter), upon node restarts, the node-id will revert to it’s standard value, causing a change of partition ownership across the cluster and fill migrations. This can further extend the upgrade procedure time if you are upgrading to version 4.2 or above, as storage needs be be erased as part of the upgrade procedure.

In order to avoid unnecessary movement of partitions you can follow the below steps while upgrading.

Solution - 2-step upgrade and set node-id

In order to make the process as smooth and fast as possible, the following steps would need to be executed, in addition to following the special upgrade instructions as you normally would:

  1. upgrade the whole cluster to the latest 3.13 (3.13.0.11 at the time of writing this article)
  2. note down node IDs of all the nodes (see below for instructions on getting node IDs)
  3. once cluster is stable, execute the special instruction protocol upgrade script, as mentioned on the 3.13 upgrade instructions page
  4. at this stage, do not reboot any nodes on version 3.13, as this would result in NODE ID change, followed by migrations. If a node goes down for any reason, continue upgrading it as per below, even if you are upgrading another node at the moment as well.
  5. take down a node (stop service)
  6. upgrade the package to the latest 4.1 as you normally would (4.1.0.6 at the time of writing this article)
  7. adjust the configuration, in particular, if you were using self-node-id and self-group-id (now known as node-id and rack-id), as follows:
    1. remove the old cluster{} stanza from the configuration file
    2. add rack-id configuration in each namespace{} stanza, with the same value that self-group-id held (if using self-group-id)
    3. add node-id configuration to the service{} stanza, setting node-id to what it was before the node was taken down (as noted in step 2)
  8. start the node
  9. once the node has joined the cluster and migrations have finished, proceed to step 4 to upgrade the next node until all nodes are upgraded
  10. once all nodes are upgraded, on 4.1, with preset node IDs, you may now upgrade over 4.2, to the desired version, ensuring you follow special upgrade instructions, in particular the 4.2 and, at the time of writing this article, 4.5.1

Getting node IDs

You can easily get node IDs of all nodes by executing the following:

$ asadm -e info
Seed:        [('127.0.0.1', 3000, None)]
Config_file: /root/.aerospike/astools.conf, /etc/aerospike/astools.conf
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information (2019-04-26 09:06:54 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             Node               Node                Ip       Build   Cluster   Migrations        Cluster     Cluster         Principal   Client     Uptime   
                .                 Id                 .           .      Size            .            Key   Integrity                 .    Conns          .   
12d9e0e140b1:3000   *BB9020011AC4202   172.17.0.2:3000   E-4.5.2.1         1      0.000     4753D6A379E1   True        BB9020011AC4202        2   00:00:23   
Number of rows: 1

Note the “Node Id” part listing the node IDs. You also get the matching hostnames/IPs under the “Node” section.

Notes

Keywords

SELF-NODE-ID SELF-GROUP-ID NODE-ID RACK-ID SPECIAL UPGRADE

Timestamp

26 April 2019

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.