Source node bb987ea7a0a0142 not found in migration rx state


I recently added two more nodes to the cluster and getting following warning in logs

source node bb987ea7a0a0142 not found in migration rx state

following is config

 # This stanza must come first.
service {
        user root
        group root
        paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
        pidfile /var/run/aerospike/
        service-threads 16
        transaction-queues 16
        transaction-threads-per-queue 8
        transaction-max-ms 10000
        transaction-pending-limit 500
        proto-fd-max 15000

logging {
        # Log file must be an absolute path.
        file /var/log/aerospike/aerospike.log {
                context any info

network {
        service {
                address any
                port 3000

        heartbeat {
                #mode multicast
                #port 9918

                # To use unicast-mesh heartbeats, comment out the 3 lines above and
                # use the following 4 lines instead.
                mode mesh
                address  # IP of the NIC on which this node is listening
                port 3002
                # IP address for seed node in the cluster
                mesh-seed-address-port 3002
                mesh-seed-address-port 3002
                mesh-seed-address-port 3002
                mesh-seed-address-port 3002
                mesh-seed-address-port 3002

                interval 150
                timeout 10

        fabric {
                port 3001

        info {
                port 3003

namespace bidder {
        replication-factor 2
        memory-size 13G
        default-ttl 90D # use 0 to never expire/evict.

        ldt-enabled true

        storage-engine device {
               device /dev/disk/by-id/google-local-ssd-0         # raw device. Maximum size is 2 TiB

               scheduler-mode noop
               write-block-size 1M   # adjust block size to make it efficient for SSDs.

               # See
               disable-odirect true


How many times does this warning appear and on how many nodes?

What version of Aerospike are you running?

We are still trying to determine how this message could appear, we don’t expect to see this warning.


So I’ve been able to determine more information about this message and I can easily reproduce it.

This message probably should be a debug level message as nothing unexpected is happening here. This message happens when a node sends a request for migration to another node containing a different version of a given partition. This isn’t necessary for every migration and may not be necessary for an entire rebalance which is why this message doesn’t come up all the time. To trigger this I need multiple unique partition versions, one of the version need to reside in the master column of the partition table.