Aerospike server was not properly switched to paxos-protocol v5


#1

After I upgrade Aerospike from aerospike-server-community-3.13.0-el7 to aerospike-server-community-3.14.0-el7, it gives me the following error during startup

Jun 08 2017 03:47:34 GMT: FAILED ASSERTION (smd): (system_metadata.c:1305) Aerospike server was not properly switched to paxos-protocol v5 - see Aerospike documentation http://www.aerospike.com/docs/operations/upgrade/cluster_to_3_13

The server is not running as part of the cluster. here is the config

service {
	user root
	group root
	paxos-protocol v5
	paxos-single-replica-limit 0 # Number of nodes where the replica count is automatically reduced to 1.
	pidfile /var/run/aerospike/asd.pid
	service-threads 4
	transaction-queues 4
	transaction-threads-per-queue 4
	proto-fd-max 100000
	proto-fd-idle-ms 15000
}

logging {
	# Log file must be an absolute path.
	file /var/log/aerospike/aerospike.log {
		context any info
	}
}

network {
	service {
		address any
		port 3000
	}

	heartbeat {		
		protocol v3                

		# To use unicast-mesh heartbeats, remove the 3 lines above, and see
		# aerospike_mesh.conf for alternative.
                mode mesh
		address 192.168.10.168
                port 3002
                mesh-seed-address-port  192.168.10.168  3002
		interval 150
		timeout 10
	}

	fabric {
		port 3001
	}

	info {
		port 3003
	}
}

here’s the asadm

[root@roomstays-a localhost ~]# asadm -e info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                  Node               Node                    Ip        Build   Cluster        Cluster     Cluster         Principal   Rackaware   Client     Uptime   
                                     .                 Id                     .            .      Size            Key   Integrity                 .        Mode    Conns          .   
roomstays-a localhost.localdomain:3000   *BB9C4C90F290C00   192.168.10.168:3000   C-3.13.0.1         1   48773DBCD75B   True        BB9C4C90F290C00   none            21   00:05:29   
Number of rows: 1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   Namespace                                     Node   Avail%   Evictions                 Master                Replica     Repl     Stop             Pending         Disk    Disk     HWM          Mem     Mem    HWM      Stop   
           .                                        .        .           .   (Objects,Tombstones)   (Objects,Tombstones)   Factor   Writes            Migrates         Used   Used%   Disk%         Used   Used%   Mem%   Writes%   
           .                                        .        .           .                      .                      .        .        .             (tx,rx)            .       .       .            .       .      .         .   
abreu_online   roomstays-a localhost.localdomain:3000   99         0.000     (0.000,  0.000)        (0.000,  0.000)        1        false    (0.000,  0.000)       0.000 B    0       50        0.000 B    0       60     90        
abreu_online                                                       0.000     (0.000,  0.000)        (0.000,  0.000)                          (0.000,  0.000)       0.000 B                      0.000 B                             
bar            roomstays-a localhost.localdomain:3000   N/E        0.000     (0.000,  0.000)        (0.000,  0.000)        1        false    (0.000,  0.000)            N/E   N/E     50        0.000 B    0       60     90        
bar                                                                0.000     (0.000,  0.000)        (0.000,  0.000)                          (0.000,  0.000)       0.000 B                      0.000 B                             
ctrip          roomstays-a localhost.localdomain:3000   99         0.000     (0.000,  0.000)        (0.000,  0.000)        1        false    (0.000,  0.000)       0.000 B    0       50        0.000 B    0       60     90        
ctrip                                                              0.000     (0.000,  0.000)        (0.000,  0.000)                          (0.000,  0.000)       0.000 B                      0.000 B                             
gallop         roomstays-a localhost.localdomain:3000   99         0.000     (0.000,  0.000)        (0.000,  0.000)        1        false    (0.000,  0.000)       0.000 B    0       50        0.000 B    0       60     90        
gallop                                                             0.000     (0.000,  0.000)        (0.000,  0.000)                          (0.000,  0.000)       0.000 B                      0.000 B                             
giata          roomstays-a localhost.localdomain:3000   96         0.000     (246.288 K, 0.000)     (0.000,  0.000)        1        false    (0.000,  0.000)      96.599 MB   3       50       15.032 MB   1       60     90        
giata                                                              0.000     (246.288 K, 0.000)     (0.000,  0.000)                          (0.000,  0.000)      96.599 MB                    15.032 MB                            
hanatour       roomstays-a localhost.localdomain:3000   99         0.000     (0.000,  0.000)        (0.000,  0.000)        1        false    (0.000,  0.000)       0.000 B    0       50        0.000 B    0       60     90        
hanatour                                                           0.000     (0.000,  0.000)        (0.000,  0.000)                          (0.000,  0.000)       0.000 B                      0.000 B                             
hotelbeds      roomstays-a localhost.localdomain:3000   71         0.000     (18.138 K, 0.000)      (0.000,  0.000)        1        false    (0.000,  0.000)     967.584 MB   24      50        1.107 MB   1       60     90        
hotelbeds                                                          0.000     (18.138 K, 0.000)      (0.000,  0.000)                          (0.000,  0.000)     967.584 MB                     1.107 MB                            
mikitravel     roomstays-a localhost.localdomain:3000   99         0.000     (57.825 K, 0.000)      (0.000,  0.000)        1        false    (0.000,  0.000)      32.865 MB   1       50        3.529 MB   1       60     90        
mikitravel                                                         0.000     (57.825 K, 0.000)      (0.000,  0.000)                          (0.000,  0.000)      32.865 MB                     3.529 MB                            
opentravel     roomstays-a localhost.localdomain:3000   99         0.000     (239.269 K, 0.000)     (0.000,  0.000)        1        false    (0.000,  0.000)      29.766 MB   1       50       14.603 MB   1       60     90        
opentravel                                                         0.000     (239.269 K, 0.000)     (0.000,  0.000)                          (0.000,  0.000)      29.766 MB                    14.603 MB                            
qunar          roomstays-a localhost.localdomain:3000   99         0.000     (2.558 K, 0.000)       (0.000,  0.000)        1        false    (0.000,  0.000)       1.703 MB   1       50      159.875 KB   1       60     90        
qunar                                                              0.000     (2.558 K, 0.000)       (0.000,  0.000)                          (0.000,  0.000)       1.703 MB                   159.875 KB                            
sunseries      roomstays-a localhost.localdomain:3000   99         0.000     (0.000,  0.000)        (0.000,  0.000)        1        false    (0.000,  0.000)       0.000 B    0       50        0.000 B    0       60     90        
sunseries                                                          0.000     (0.000,  0.000)        (0.000,  0.000)                          (0.000,  0.000)       0.000 B                      0.000 B                             
test           roomstays-a localhost.localdomain:3000   N/E        0.000     (0.000,  0.000)        (0.000,  0.000)        1        false    (0.000,  0.000)            N/E   N/E     50        0.000 B    0       60     90        
test                                                               0.000     (0.000,  0.000)        (0.000,  0.000)                          (0.000,  0.000)       0.000 B                      0.000 B                             
uapi           roomstays-a localhost.localdomain:3000   N/E        0.000     (0.000,  0.000)        (0.000,  0.000)        1        false    (0.000,  0.000)            N/E   N/E     50        0.000 B    0       60     90        
uapi                                                               0.000     (0.000,  0.000)        (0.000,  0.000)                          (0.000,  0.000)       0.000 B                      0.000 B                             
Number of rows: 26

#2

Did you change to paxos-protocol v5 while running 3.13? The switch migrates sindex related smd files. The conversion is only supoorted in 3.13.

You may either:

  1. Reinstall 3.13 and start the server. This will run the conversions.
  2. Remove sindex related smd files located at (iirc) /opt/aerospike/smd/. Then start server and recreate secondary indices.

#3

This works, thanks