# Stuck in adding nodes

**URL:** <https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999>\
**Category:** Operations\
**Created:** [March 8, 2018, 8:26pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999 "2018-03-08T20:26:07Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 8:26pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/1 "2018-03-08T20:26:07Z")

</div>

Am in 3.9.0.2.

I added 5 new empty nodes to existing 5 nodes with data. It is not joining the cluster.

```auto
asadm -> Cluster Visibility error (Please check services list):

```

```auto
Mar 08 2018 20:15:13 GMT: INFO (info): (ticker.c:415) {seooi} device-usage: used-bytes 0 avail-pct 99
Mar 08 2018 20:15:13 GMT: INFO (partition): (partition.c:235) DISALLOW MIGRATIONS
Mar 08 2018 20:15:13 GMT: INFO (paxos): (paxos.c:147) cluster_key set to 0x799ae23ce8966716
Mar 08 2018 20:15:13 GMT: INFO (paxos): (paxos.c:3201) SUCCESSION [1520540107]@bb998c054005452*: bb998c054005452 bb997c054005452 bb996c054005452 bb995c054005452 bb994c054005452 bb94bc354005452 bb94ac354005452 bb949c354005452 bb948c354005452 bb946c354005452 
Mar 08 2018 20:15:13 GMT: INFO (paxos): (paxos.c:3212) node bb998c054005452 is still principal pro tempore
Mar 08 2018 20:15:13 GMT: INFO (paxos): (paxos.c:2328) Sent partition sync request to node bb998c054005452

```

I see ports are pinging to and fro on both for 300{1,2,3} for both the clusters.

At least I would like to halt adding nodes. And let old nodes to live. Am afraid of below details on new nodes only. Pending migrates is stuck like this. Not sure if I can shutdown new guys. Need help as the apps are asleep now.

```
 Node Avail% Evictions Master Replica Repl Stop Pending 
    . . . Objects Objects Factor Writes Migrates 
    . . . . . . . (tx%,rx%) 
:3000 99 0.000 0.000 0.000 2 false (0,0)     
:3000 99 0.000 0.000 0.000 2 false (0,0)     
:3000 99 0.000 0.000 0.000 2 false (100,100) 
:3000 99 0.000 0.000 0.000 2 false (100,100) 
:3000 99 0.000 0.000 0.000 2 false (0,0)     
                   0.000 0.000 0.000 (31,52)

```

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [March 8, 2018, 9:04pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/2 "2018-03-08T21:04:54Z")

</div>

Could you show the rest of the output from `asadm -e info`?

Could you also run:

```auto
asinfo -e 'show stat like migrate'

```

---

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 9:11pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/3 "2018-03-08T21:11:34Z")

</div>

```auto
Admin> show stat like migrate
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Service Statistics~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE : survey001.com:3000 survey002.com:3000 survey003.com:3000 survey004.com:3000 survey005.com:3000  
migrate_allowed : false false false false false                                                      
migrate_partitions_remaining: 0 0 2720 2709 0                                                          
migrate_progress_recv : 0 0 2720 2709 0                                                          
migrate_progress_send : 0 0 2720 2709 0                                                          

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Statistics~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE : survey001.com:3000 survey002.com:3000 survey003.com:3000 survey004.com:3000 survey005.com:3000  
migrate-order : 5 5 5 5 5                                                          
migrate-sleep : 1 1 1 1 1                                                          
migrate_record_receives : 0 0 0 0 0                                                          
migrate_record_retransmits : 0 0 0 0 0                                                          
migrate_records_skipped : 0 0 0 0 0                                                          
migrate_records_transmitted : 0 0 0 0 0                                                          
migrate_rx_instances : 0 0 0 0 0                                                          
migrate_rx_partitions_active : 0 0 0 0 0                                                          
migrate_rx_partitions_initial : 825 859 1904 1899 1956                                                       
migrate_rx_partitions_remaining: 0 0 1904 1899 0                                                          
migrate_tx_instances : 0 0 0 0 0                                                          
migrate_tx_partitions_active : 0 0 0 0 0                                                          
migrate_tx_partitions_imbalance: 0 0 0 0 0                                                          
migrate_tx_partitions_initial : 1392 1412 816 810 836                                                        
migrate_tx_partitions_remaining: 0 0 816 810 0                                                          

Admin> 
>> 

survey001.com:3000 BB946C354005452 11.14.11.1:3000 C-3.9.0.2 3 E5AA55F4E5F82B35 True BB998C054005452 16 01:30:32   
survey002.com:3000 BB948C354005452 11.14.11.2:3000 C-3.9.0.2 3 E5AA55F4E5F82B35 True BB998C054005452 13 01:30:32   
survey003.com:3000 BB949C354005452 11.14.11.3:3000 C-3.9.0.2 3 E5AA55F4E5F82B35 True BB998C054005452 16 01:30:32   
survey004.com:3000 BB94AC354005452 11.14.11.4:3000 C-3.9.0.2 3 E5AA55F4E5F82B35 True BB998C054005452 16 01:30:32   
survey005.com:3000 *BB94BC354005452 11.14.11.5:3000 C-3.9.0.2 3 E5AA55F4E5F82B35 True BB998C054005452 17 01:30:32   
Number of rows: 5

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Namespace Node Avail% Evictions Master Replica Repl Stop Pending Disk Disk HWM Mem Mem HWM Stop   
          . . . . Objects Objects Factor Writes Migrates Used Used% Disk% Used Used% Mem% Writes%   
          . . . . . . . . (tx%,rx%) . . . . . . .   
seooi 000000001.com:3000 99 0.000 0.000 0.000 2 false (0,0) 0.000 B 0 50 0.000 B 0 60 90        
seooi 000000002.com:3000 99 0.000 0.000 0.000 2 false (0,0) 0.000 B 0 50 0.000 B 0 60 90        
seooi 000000003.com:3000 99 0.000 0.000 0.000 2 false (100,100) 0.000 B 0 50 0.000 B 0 60 90        
seooi 000000004.com:3000 99 0.000 0.000 0.000 2 false (100,100) 0.000 B 0 50 0.000 B 0 60 90        
seooi 000000005.com:3000 99 0.000 0.000 0.000 2 false (0,0) 0.000 B 0 50 0.000 B 0 60 90        
seooi 0.000 0.000 0.000 (31,52) 0.000 B 0.000 B                             
Number of rows: 6

```

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [March 8, 2018, 9:17pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/4 "2018-03-08T21:17:20Z")

</div>

Are your namespaces in the same position in the aerospike.conf on the newer nodes?

Is this a mesh or multicast configured cluster?

---

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 9:21pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/5 "2018-03-08T21:21:10Z")

</div>

> [@lucien](#):
>
> Are your namespaces in the same position in the aerospike.conf on the newer nodes?

Older ones don’t have new nodes IPs and new ones has (older one & new node) IPs. Other than that rest are the same. Also older one has another NS, that is not required on new nodes. So ignored that namespace block in new nodes.

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [March 8, 2018, 9:24pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/6 "2018-03-08T21:24:49Z")

</div>

> [@Mannoj](#):
>
> Also older one has another NS, that is not required on new nodes. So ignored that namespace block in new nodes.

That is the issue! On the older version the namespace count and position have to match!

---

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 9:25pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/7 "2018-03-08T21:25:18Z")

</div>

You mean I should have all Namespaces to new ones?

---

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 9:27pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/8 "2018-03-08T21:27:29Z")

</div>

> [@Mannoj](#):
>
> You mean I should have all Namespaces to new ones?

If that is the case. Can I shut down now? And re-do it with adding left out namespace as well? Is the new cluster is in safe state? I will take backup of old. Then rm -f dat file on new guys and then add entries of both NS and then start new nodes. Is this approach is good?

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [March 8, 2018, 9:27pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/9 "2018-03-08T21:27:55Z")

</div>

Correct! namespace position and namepace number have to be the same. I the version you have adding a namespace requires a cluster shutdown. This has changed in newer versions.

I think the steps could be ,

-Take a backup.

-Shutdown the cluster

-Fix config to be similar in terms of namespaces

-Restart the cluster.

I don’t believe you need to rm any storage files.

---

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 9:32pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/10 "2018-03-08T21:32:32Z")

</div>

-Take a backup. (FROM Current AS data nodes)

-Shutdown the cluster (On NEW nodata nodes)

-Fix config to be similar in terms of namespaces (On NEW nodata nodes)

-Restart the cluster. (On NEW nodata nodes)

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [March 8, 2018, 9:34pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/11 "2018-03-08T21:34:47Z")

</div>

correct! that should work! Backup is not really needed, but its always good practice!

---

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 9:44pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/12 "2018-03-08T21:44:48Z")

</div>

While running backup - 2018-03-08 21:38:40 GMT [ERR] [15949] Error while running node scan for BB998C054005452 - code 7: AEROSPIKE\_ERR\_CLUSTER\_CHANGE at src/main/aerospike/aerospike\_scan.c:190

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [March 8, 2018, 9:52pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/13 "2018-03-08T21:52:20Z")

</div>

Ah yes you are doing a backup during a cluster change.

Please see:

[https://discuss.aerospike.com/t/aerospike-err-cluster-change-when-querying-or-scanning-a-namespace/2593](https://discuss.aerospike.com/t/aerospike-err-cluster-change-when-querying-or-scanning-a-namespace/2593)

asbackup does have a flag that allows for this.

---

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 10:03pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/14 "2018-03-08T22:03:00Z")

</div>

I removed the arguement - --no-cluster-change . Its going on. Do you think apps are not responding to their GETS at this moment?

---

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 10:26pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/15 "2018-03-08T22:26:55Z")

</div>

Above steps are done. Migrations are kicking in . Happy to see myself now. 😉 Thanks all, you guys are really helpful when needed.

---

<div class="post-metadata">

**Author:** ![Mannoj](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@Mannoj](https://discuss.aerospike.com/u/Mannoj)\
**Post date:** [March 8, 2018, 10:46pm UTC](https://discuss.aerospike.com/t/stuck-in-adding-nodes/4999/16 "2018-03-08T22:46:50Z")

</div>

> [@lucien](#):
>
> older version the namespace count and position have to match!

I see that from [3.13.3 onwards](https://www.aerospike.com/blog/whats-new-aerospike-3-13-3-14/) it supports certain nodes only to have one namespace.
