Migration of Aerospike cluster without downtime

Operations
22

@kporter @Albot I have changed the address from any to specific Ip and restart the server, but still new node is not able to connect with cluster.

Below is the aerospike.conf of the existing machine in cluster.

network {
        service {
                address 172.20.21.185
                port 3000
                access-address 172.20.21.185
        }

        heartbeat {
                mode mesh
                address 172.20.21.185
                port 3002
                # To use unicast-mesh heartbeats, remove the 3 lines above, and see
                # aerospike_mesh.conf for alternative.

                mesh-seed-address-port 172.20.21.186 3002
                mesh-seed-address-port 172.20.21.187 3002
                mesh-seed-address-port 172.20.21.188 3002
                mesh-seed-address-port 172.20.21.192 3002


                interval 150
                timeout 10
        }

        fabric {
                address 172.20.21.185
                port 3001
        }

        info {
                address 172.20.21.185
                port 3003            
             }
}

But I am getting below in logs when I restarted this machine.

Jul 20 2017 03:45:33 GMT+0530: INFO (as): (as.c:468) initializing services...
Jul 20 2017 03:45:33 GMT+0530: INFO (tsvc): (thr_tsvc.c:473) shared queues: 4 queues with 4 threads each
Jul 20 2017 03:45:33 GMT+0530: INFO (drv_ssd): (drv_ssd.c:1475) ns lgp starting write worker threads
Jul 20 2017 03:45:33 GMT+0530: INFO (drv_ssd): (drv_ssd.c:910) ns lgp starting defrag threads
Jul 20 2017 03:45:33 GMT+0530: INFO (hb): (hb.c:3132) Using 172.20.21.185 as the IP address for receiving heartbeats
Jul 20 2017 03:45:33 GMT+0530: INFO (hb): (hb.c:3194) Using 10.40.0.151:3002 as the published IP address
Jul 20 2017 03:45:33 GMT+0530: INFO (hb): (hb.c:1645) Added mesh seed node from config 172.20.21.186:3002
Jul 20 2017 03:45:33 GMT+0530: INFO (hb): (hb.c:1645) Added mesh seed node from config 172.20.21.187:3002
Jul 20 2017 03:45:33 GMT+0530: INFO (hb): (hb.c:1645) Added mesh seed node from config 172.20.21.188:3002

Not sure why published Ip address is different from IP which I have mentioned in config file.

Below are the logs from new machine, which I am trying to add in existing cluster.

Jul 20 2017 12:00:00 GMT+0530: INFO (hb): (hb.c:6365) Updating mesh seed endpoint address from 172.20.21.185:3002 to 10.40.0.151:3002
Jul 20 2017 12:00:00 GMT+0530: INFO (hb): (hb.c:6365) Updating mesh seed endpoint address from 172.20.21.188:3002 to 10.40.0.154:3002
Jul 20 2017 12:00:00 GMT+0530: INFO (hb): (hb.c:6365) Updating mesh seed endpoint address from 172.20.21.186:3002 to 10.40.0.152:3002
Jul 20 2017 12:00:00 GMT+0530: INFO (hb): (hb.c:6365) Updating mesh seed endpoint address from 172.20.21.187:3002 to 10.40.0.153:3002
Jul 20 2017 12:00:00 GMT+0530: INFO (fabric): (fabric.c:1378) fabric: node bb9245fbd565000 arrived
Jul 20 2017 12:00:00 GMT+0530: INFO (fabric): (fabric.c:1378) fabric: node bb92e0fbd565000 arrived
Jul 20 2017 12:00:00 GMT+0530: INFO (fabric): (fabric.c:1378) fabric: node bb94179bd565000 arrived
Jul 20 2017 12:00:00 GMT+0530: INFO (fabric): (fabric.c:1378) fabric: node bb99214bd565000 arrived
Jul 20 2017 12:00:00 GMT+0530: INFO (fabric): (fabric.c:1378) fabric: node bb9ef48bd565000 arrived
Jul 20 2017 12:00:01 GMT+0530: INFO (paxos): (paxos.c:3792) ... other node(s) detected - node will operate in a multi-node cluster
Jul 20 2017 12:00:01 GMT+0530: INFO (paxos): (paxos.c:3730) paxos supervisor thread started
Jul 20 2017 12:00:01 GMT+0530: INFO (paxos): (paxos.c:2263) Skip node arrival bb9245fbd565000 cluster principal bb999f5540a0102 pulse principal bb9ef48bd565000
Jul 20 2017 12:00:01 GMT+0530: INFO (paxos): (paxos.c:2263) Skip node arrival bb92e0fbd565000 cluster principal bb999f5540a0102 pulse principal bb9ef48bd565000
Jul 20 2017 12:00:01 GMT+0530: INFO (paxos): (paxos.c:2263) Skip node arrival bb94179bd565000 cluster principal bb999f5540a0102 pulse principal bb9ef48bd565000
Jul 20 2017 12:00:01 GMT+0530: INFO (paxos): (paxos.c:2263) Skip node arrival bb99214bd565000 cluster principal bb999f5540a0102 pulse principal bb9ef48bd565000
Jul 20 2017 12:00:01 GMT+0530: INFO (paxos): (paxos.c:2263) Skip node arrival bb9ef48bd565000 cluster principal bb999f5540a0102 pulse principal bb9ef48bd565000

It seems that new node got cluster information but couldn’t join cluster from logs. My main question is, what should be the config to bind all process heartbeat,fabric, info to speacif IP. I have attached my config file above, do let me know if any change is required in this.

Thanks in advance

23

I believe you are looking for interface-address.

24

@Albot @kporter Thanks for the help guys. really appreciated. Issue is resolved.

interface-address and network-interface-name were the keys for this issue. added these in config fine and was able to add new machine in cluster.