The cluster stops working when adding a new node

I have the cluster with 2 nodes ( version) . I need to add 2 new nodes. I have installed 2 nodes (4.1.7 version). When I start new node reads and writes in AMC drops down to 0, my application writes error:

Timeout: iterations=0 lastNode= in src/main/aerospike/as_event.c:467

Configs are same on all nodes:

service {
    paxos-single-replica-limit 1
    proto-fd-max 15000

logging {
    file /var/log/aerospike/aerospike.log {
        context any info

network {
    service {
        address any
        port 3000

    heartbeat {
        mode mesh
        address [IP of this node]
        port 3002

        mesh-seed-address-port [IP of this node] 3002
        mesh-seed-address-port [IP of node 2] 3002
        mesh-seed-address-port [IP of node 3] 3002
        mesh-seed-address-port [IP of node 4] 3002

        interval 150
        timeout 10

    fabric {
        port 3001

    info {
        port 3003

namespace mem {
    replication-factor 2
    memory-size 48G
    default-ttl 1d

    storage-engine device {
        file /data/aerospike/mem.dat
        filesize 64G
        data-in-memory true

All tcp communications are allowed between nodes… In logs I see new node added (NODE-ID bb9dec1da0e1b90 CLUSTER-SIZE 3), migration started. And only warnings about connection to node 4 (which is stopped).

Then I stop the node 3 and there are operations in AMC as if nothing had happened…

Why this can happen?

You mentioned the application stops connecting, but only tcp connectivity works between nodes… Does tcp connectivity work between app and new nodes? Are you using the Aerospike logging interface? What happens if you run ‘asadm -e health’ ?

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.