2019-06-12 09:03:06 GMT [INF] [12619] Processing 3 node(s)
2019-06-12 09:03:06 GMT [INF] [12619] Node ID Objects Replication
2019-06-12 09:03:06 GMT [INF] [12619] BB9552D200A0102 168623562 2
2019-06-12 09:03:06 GMT [INF] [12619] BB99193210A0102 171760439 2
2019-06-12 09:03:06 GMT [INF] [12619] BB94735220A0102 171441109 2
2019-06-12 09:03:06 GMT [INF] [12619] Namespace contains 255912555 record(s)
2019-06-12 09:03:06 GMT [INF] [12619] Directory /opt/backup/ugc-aerospike does not exist, creating
2019-06-12 09:03:06 GMT [INF] [12619] Directory /opt/backup/ugc-aerospike prepared for backup
2019-06-12 09:03:06 GMT [INF] [12638] Starting backup for node BB9552D200A0102
2019-06-12 09:03:06 GMT [INF] [12640] Starting backup for node BB94735220A0102
2019-06-12 09:03:06 GMT [INF] [12639] Starting backup for node BB99193210A0102
2019-06-12 09:03:06 GMT [INF] [12639] Created new backup file /opt/backup/ugc-aerospike/BB99193210A0102_00000.asb
2019-06-12 09:03:06 GMT [INF] [12638] Created new backup file /opt/backup/ugc-aerospike/BB9552D200A0102_00000.asb
2019-06-12 09:03:06 GMT [INF] [12640] Created new backup file /opt/backup/ugc-aerospike/BB94735220A0102_00000.asb
2019-06-12 09:03:06 GMT [INF] [12638] Backing up 4 secondary index(es)
2019-06-12 09:03:06 GMT [INF] [12638] Backing up 3 UDF file(s)
2019-06-12 09:03:07 GMT [INF] [12637] 0% complete (~307859 KiB/s, ~430138 rec/s, ~732 B/rec)
2019-06-12 09:03:07 GMT [INF] [12637] ~9m53s remaining
2019-06-12 09:03:13 GMT [INF] [12640] Created new backup file /opt/backup/ugc-aerospike/BB94735220A0102_00001.asb
2019-06-12 09:03:14 GMT [INF] [12638] Created new backup file /opt/backup/ugc-aerospike/BB9552D200A0102_00001.asb
2019-06-12 09:03:14 GMT [INF] [12639] Created new backup file /opt/backup/ugc-aerospike/BB99193210A0102_00001.asb
2019-06-12 09:03:14 GMT [ERR] [12639] Error while running node scan for BB99193210A0102 - code 7: AEROSPIKE_ERR_CLUSTER_CHANGE at src/main/aerospike/aerospike_scan.c:197
2019-06-12 09:03:14 GMT [INF] [12638] Node scan for BB9552D200A0102 aborted
2019-06-12 09:03:14 GMT [INF] [12640] Node scan for BB94735220A0102 aborted
I have verified using asadm that migrations aren’t happening.
Albot
2
Did the cluster-key change? If this is easy to reproduce, try grabbing an asadm info snapshot before/after and compare the cluster key…
Cluster key is changing continuously when ran asadm info command multiple times.
Admin> info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Ip Build Cluster Cluster Cluster Principal Rackaware Client Uptime
. Id . . Size Key Integrity . Mode Conns .
10.33.147.145:3000 *BB99193210A0102 10.33.147.145:3000 C-3.12.0 3 51242B64B81D0AFB True BB9B43D200A0102 none 1312 210:49:48
10.34.53.71:3000 BB94735220A0102 10.34.53.71:3000 C-3.12.0 3 51242B64B81D0AFB True BB9B43D200A0102 none 1273 214:29:35
ugc-aerospike-prod-3-1043017:3000 BB9552D200A0102 10.32.45.85:3000 C-3.12.0 3 51242B64B81D0AFB True BB9B43D200A0102 none 1296 225:18:41
Number of rows: 3
Admin> info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Ip Build Cluster Cluster Cluster Principal Rackaware Client Uptime
. Id . . Size Key Integrity . Mode Conns .
10.33.147.145:3000 *BB99193210A0102 10.33.147.145:3000 C-3.12.0 3 982CA5117EB01572 True BB9B43D200A0102 none 1313 210:49:52
10.34.53.71:3000 BB94735220A0102 10.34.53.71:3000 C-3.12.0 3 982CA5117EB01572 True BB9B43D200A0102 none 1273 214:29:39
ugc-aerospike-prod-3-1043017:3000 BB9552D200A0102 10.32.45.85:3000 C-3.12.0 3 982CA5117EB01572 True BB9B43D200A0102 none 1296 225:18:45
Number of rows: 3
Admin> info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Ip Build Cluster Cluster Cluster Principal Rackaware Client Uptime
. Id . . Size Key Integrity . Mode Conns .
10.33.147.145:3000 *BB99193210A0102 10.33.147.145:3000 C-3.12.0 3 38D61BE8BF4CB035 True BB9B43D200A0102 none 1312 210:51:02
10.34.53.71:3000 BB94735220A0102 10.34.53.71:3000 C-3.12.0 3 38D61BE8BF4CB035 True BB9B43D200A0102 none 1273 214:30:49
ugc-aerospike-prod-3-1043017:3000 BB9552D200A0102 10.32.45.85:3000 C-3.12.0 3 38D61BE8BF4CB035 True BB9B43D200A0102 none 1299 225:19:55
Number of rows: 3
Albot
4
So your cluster keeps reforming. Something is wrong. You should see something in the logs indicating why