Source node bb987ea7a0a0142 not found in migration rx state

1

I recently added two more nodes to the cluster and getting following warning in logs

source node bb987ea7a0a0142 not found in migration rx state

following is config

 # This stanza must come first.
service {
        user root
        group root
        paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
        pidfile /var/run/aerospike/asd.pid
        service-threads 16
        transaction-queues 16
        transaction-threads-per-queue 8
        transaction-max-ms 10000
        transaction-pending-limit 500
        proto-fd-max 15000
}

logging {
        # Log file must be an absolute path.
        file /var/log/aerospike/aerospike.log {
                context any info
        }
}


network {
        service {
                address any
                port 3000
        }

        heartbeat {
                #mode multicast
                #address 239.1.99.222
                #port 9918

                # To use unicast-mesh heartbeats, comment out the 3 lines above and
                # use the following 4 lines instead.
                mode mesh
                address 10.147.223.143  # IP of the NIC on which this node is listening
                port 3002
                # IP address for seed node in the cluster
                mesh-seed-address-port 10.59.30.16 3002
                mesh-seed-address-port 10.81.51.139 3002
                mesh-seed-address-port 10.122.234.135 3002
                mesh-seed-address-port 10.103.249.246 3002
                mesh-seed-address-port 10.147.223.143 3002

                interval 150
                timeout 10
        }

        fabric {
                port 3001
        }

        info {
                port 3003
        }
}

namespace bidder {
        replication-factor 2
        memory-size 13G
        default-ttl 90D # use 0 to never expire/evict.

        ldt-enabled true

        storage-engine device {
               device /dev/disk/by-id/google-local-ssd-0         # raw device. Maximum size is 2 TiB

               scheduler-mode noop
               write-block-size 1M   # adjust block size to make it efficient for SSDs.

               # See http://discuss.aerospike.com/t/aerospike-doesnt-work-with-local-ssd-disks-at-gce/497/4
               disable-odirect true
       }
}
2

How many times does this warning appear and on how many nodes?

What version of Aerospike are you running?

We are still trying to determine how this message could appear, we don’t expect to see this warning.

3

So I’ve been able to determine more information about this message and I can easily reproduce it.

This message probably should be a debug level message as nothing unexpected is happening here. This message happens when a node sends a request for migration to another node containing a different version of a given partition. This isn’t necessary for every migration and may not be necessary for an entire rebalance which is why this message doesn’t come up all the time. To trigger this I need multiple unique partition versions, one of the version need to reside in the master column of the partition table.