# How to enable rack aware on an existing cluster: best practices

**URL:** <https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822>\
**Category:** Configuration\
**Created:** [September 10, 2015, 3:36pm UTC](https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822 "2015-09-10T15:36:06Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![naoum](https://avatars.discourse-cdn.com/v4/letter/n/4da419/32.png) [@naoum](https://discuss.aerospike.com/u/naoum)\
**Post date:** [September 10, 2015, 3:36pm UTC](https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822/1 "2015-09-10T15:36:06Z")

</div>

Hi,

I have a cluster with 4 nodes that I want to enable rack awareness on. Currently they are all on the same rack so as part of the process I need to replace 2 of the nodes with nodes on a different rack. I also need to change the config and restart each node as per [http://www.aerospike.com/docs/operations/configure/network/rack-aware/](http://www.aerospike.com/docs/operations/configure/network/rack-aware/) . What is the recommended sequence to do all that? I remember seeing a weird situation on my test cluster where it was complaining that the cluster is was not balanced when I enabled rack aware there so I want to make sure I don’t get into a similar situation. I also want to minimize the number of migrations as each one can take a while.

Thanks!

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [September 11, 2015, 10:15pm UTC](https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822/2 "2015-09-11T22:15:58Z")

</div>

Are you bringing down 2 nodes and adding 2 fresh nodes on a different rack?

---

<div class="post-metadata">

**Author:** ![naoum](https://avatars.discourse-cdn.com/v4/letter/n/4da419/32.png) [@naoum](https://discuss.aerospike.com/u/naoum)\
**Post date:** [September 11, 2015, 10:16pm UTC](https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822/3 "2015-09-11T22:16:48Z")

</div>

@kporter yes, that’s the plan

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [September 11, 2015, 10:40pm UTC](https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822/5 "2015-09-11T22:40:27Z")

</div>

# Assuming you cannot have downtime

**Procedure removed due to problems described in my next post.**

# If you are able to have downtime,

1. Configure your existing servers into two racks (the nodes that are staying and the nodes that are leaving).
2. Configure the new servers into a third rack.
3. Stop all nodes
4. Start all nodes
5. Wait for all mirations to complete
6. Drop the nodes (since they are a rack it will be safe to drop them both at the same time).

---

<div class="post-metadata">

**Author:** ![naoum](https://avatars.discourse-cdn.com/v4/letter/n/4da419/32.png) [@naoum](https://discuss.aerospike.com/u/naoum)\
**Post date:** [September 15, 2015, 5:36pm UTC](https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822/6 "2015-09-15T17:36:23Z")

</div>

@kporter,

Thanks a lot for your response! I am definitely in the “Assuming you cannot have downtime” case so a couple of clarifying questions:

- In Step 4) when I restart the nodes this will pick up the rack groups and also cause each node to re-join the cluster and migrate all the data back to it

- I just tried setting the heartbeat protocol to none and got the following error - is that only needed in mesh setup?

```
Sep 15 2015 23:52:25 GMT: INFO (info): (thr_info.c::3271) Changing value of heartbeat protocol version to none 
Sep 15 2015 23:52:25 GMT: WARNING (hb): (hb.c::1384) setting heartbeat protocol is only supported in heartbeat mode "multicast"
```

Thanks!

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [September 16, 2015, 1:30am UTC](https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822/7 "2015-09-16T01:30:23Z")

</div>

Currently only the multicast protocol supports changing protocols, so there will not be a way to do this without downtime.

😊 Actually after discussing this I also learned that the procedure would have had problems; unrestarted nodes in the cluster while doing the restarts would be able to find much of the data and the replication would be directed to the wrong nodes by the unrestarted nodes.

---

<div class="post-metadata">

**Author:** ![naoum](https://avatars.discourse-cdn.com/v4/letter/n/4da419/32.png) [@naoum](https://discuss.aerospike.com/u/naoum)\
**Post date:** [September 16, 2015, 5:54pm UTC](https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822/8 "2015-09-16T17:54:45Z")

</div>

@kporter,

So in the downtime approach – is there a step 0) to bring down all the servers first? If I am restarting servers in step 3) is it safe to have some that think they are in a rack while others who don’t?

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [September 16, 2015, 7:46pm UTC](https://discuss.aerospike.com/t/how-to-enable-rack-aware-on-an-existing-cluster-best-practices/1822/9 "2015-09-16T19:46:26Z")

</div>

Updated the procedure to clarify.
