Backup Stay at Specific Percentage!

Tools asbackup & asrestore
1

I have 15 node Aerospike running on 3.5.8 Enterprise edition and store 2.5TB on SSD with 500GB on memory. We only have one namespace with 10 sets. I tried to backup our data multiple times with different parameters but each time backup process keep hanging in some point and couldn’t finalize the backup even after days!

At end, I tried to backup one server at a time with following command:

asbackup --l "10.53.1.1 --namespace BlueCava --directory /data/aerospike/server-01 -c -f 3 --verbose

Which 3 of servers completed 100% and most of them stopped at 20-25% and some at 75-80%! How can I debug what’s going on? Is there better way to backup our data?

2

The scan implementation which asbackup is built on was re-architected in 3.6.X. There were some conditions under which the old implementation of scan would get locked up, especially when a lot of concurrent scans were issued. I would recommend upgrading to the latest version and try again. You can also check the statistics for the number of scans pending tscan_pending to confirm whether there is a chance of hitting such issue.

1 Like
3

Thanks @meher for your reply :slight_smile:

We came to same conclusion. We found more than 60k running scans that stuck and prevent backup to complete it’s process. BTW, is there any easy way to kill all these scans?

I finally decided to upgrade my cluster to 3.7.4.1 which released Today! with XDR issue fixed. I’ll re-run backup as soon as upgrade process finalized. Also I’m happy to use new parameters like --compact and --nice for asbackup :wink:

4

Unfortunately I don’t think that’s possible on 3.5.8… For 3.5.12 and above, to kill all on-going scans, use the following syntax:

asinfo -v 'scan-abort-all:'
OK - number of scans killed: 134
5

Finally upgrade done and I’m running all nodes on 3.7.4.1! I run backup with this parameters:

    asbackup \
        --host 10.53.1.1 \
        --port 3000 \
        --namespace MyNameSpace \
        --set MySet \
        --directory /data/MySet \
        --no-cluster-change \
        --parallel 15 \
        --priority 3 \
        --verbose

Backup process look like completed successfully. However when it was on 15% suddenly it said backup completed without any error and returned status “0”!

Is this normal? Is that percentage is just estimation which might not be accurate? Although 85% is too big difference to be estimate!

6

This is full Verbose output for asbackup. This is currently running latest version (3.7.4.1) - PART 1:

2016-03-13 03:11:35 GMT [INF] [15016] Starting 100% backup of 10.53.1.1:3000 (namespace: BlueCava, set: SnapshotHashes, bins: [all]) to /data/aerospike/sets/SnapshotHashes/all
2016-03-13 03:11:35 GMT [VER] [15016] Connecting to cluster
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:400][as_cluster_seed_nodes] Add node BB98C6632679DD8 10.53.1.1:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB98C6632679DD8 partition generation changed: 789
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB9A81A2B679DD8 10.53.1.7:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB908C52B679DD8 10.53.1.15:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB90C6B32679DD8 10.53.1.12:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB9107C32679DD8 10.53.1.13:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB914C72A679DD8 10.53.1.11:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB92C7C32679DD8 10.53.1.4:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB938B832679DD8 10.53.1.5:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB9443D32679DD8 10.53.1.10:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB950B832679DD8 10.53.1.3:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB958B832679DD8 10.53.1.6:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB988B532679DD8 10.53.1.2:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB990512B679DD8 10.53.1.14:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB990862B679DD8 10.53.1.8:3000
2016-03-13 03:11:35 GMT [INF] [15016] [src/main/aerospike/as_cluster.c:468][as_cluster_find_nodes_to_add] Add node BB9A0B432679DD8 10.53.1.9:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB9A81A2B679DD8 partition generation changed: 1177
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB908C52B679DD8 partition generation changed: 1925
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB9A81A2B679DD8 partition generation changed: 1177
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB908C52B679DD8 partition generation changed: 1925
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB90C6B32679DD8 partition generation changed: 1491
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB9107C32679DD8 partition generation changed: 1564
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB914C72A679DD8 partition generation changed: 1611
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB92C7C32679DD8 partition generation changed: 1143
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB938B832679DD8 partition generation changed: 1312
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB9443D32679DD8 partition generation changed: 1435
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB950B832679DD8 partition generation changed: 1070
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB958B832679DD8 partition generation changed: 922
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB988B532679DD8 partition generation changed: 966
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB990512B679DD8 partition generation changed: 1745
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB990862B679DD8 partition generation changed: 1361
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_node.c:543][as_node_process_response] Node BB9A0B432679DD8 partition generation changed: 1361
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:889][as_cluster_add_seeds] Add seed 10.53.1.1:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.7:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.15:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.12:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.13:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.11:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.4:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.5:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.10:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.3:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.6:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.2:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.14:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.8:3000
2016-03-13 03:11:35 GMT [VER] [15016] [src/main/aerospike/as_cluster.c:140][as_seeds_add] Add seed 10.53.1.9:3000
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB98C6632679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB9A81A2B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB908C52B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB9A81A2B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB908C52B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB90C6B32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB9107C32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB914C72A679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB92C7C32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB938B832679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB9443D32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB950B832679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB958B832679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB988B532679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB990512B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB990862B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Adding node BB9A0B432679DD8
2016-03-13 03:11:35 GMT [INF] [15016] Processing 15 node(s)
2016-03-13 03:11:35 GMT [VER] [15016] Getting cluster object count
2016-03-13 03:11:35 GMT [INF] [15016] Node ID             Objects        Replication
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB98C6632679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB98C6632679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB98C6632679DD8     442423377      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB9A81A2B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB9A81A2B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB9A81A2B679DD8     442814069      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB908C52B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB908C52B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB908C52B679DD8     430354271      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB90C6B32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB90C6B32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB90C6B32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB90C6B32679DD8     430492298      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB9107C32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB9107C32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB9107C32679DD8     431353961      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB914C72A679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB914C72A679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB914C72A679DD8     442767337      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB92C7C32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB92C7C32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB92C7C32679DD8     442276884      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB938B832679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB938B832679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB938B832679DD8     442834468      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB9443D32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB9443D32679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB9443D32679DD8     442557400      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB950B832679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB950B832679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB950B832679DD8     442462001      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB958B832679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB958B832679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB958B832679DD8     442852801      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB988B532679DD8
2016-03-13 03:11:35 GMT [INF] [15016] BB958B832679DD8     442852801      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB988B532679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB988B532679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB988B532679DD8     442908803      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB990512B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB990512B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB990512B679DD8     426163365      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB990862B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB990862B679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB990862B679DD8     442438962      2
2016-03-13 03:11:35 GMT [VER] [15016] Getting object count for node BB9A0B432679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Getting info value namespace/BlueCava for node BB9A0B432679DD8
2016-03-13 03:11:35 GMT [VER] [15016] Parsing info
2016-03-13 03:11:35 GMT [INF] [15016] BB9A0B432679DD8     442204901      2
2016-03-13 03:11:35 GMT [INF] [15016] Namespace contains 3293452449 record(s)
2016-03-13 03:11:35 GMT [VER] [15016] Preparing backup directory /data/aerospike/sets/SnapshotHashes/all
2016-03-13 03:11:35 GMT [INF] [15016] Directory /data/aerospike/sets/SnapshotHashes/all prepared for backup
2016-03-13 03:11:35 GMT [VER] [15016] Creating counter thread
2016-03-13 03:11:35 GMT [VER] [15016] Pushing 15 job(s) to job queue
2016-03-13 03:11:35 GMT [VER] [15016] Creating 15 backup thread(s)
2016-03-13 03:11:35 GMT [VER] [15016] Waiting for 15 backup thread(s)
2016-03-13 03:11:35 GMT [VER] [15037] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15037] Starting backup for node BB98C6632679DD8
2016-03-13 03:11:35 GMT [VER] [15037] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15037] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB98C6632679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15037] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15037] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15037] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB98C6632679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15037] Initializing backup file
2016-03-13 03:11:35 GMT [INF] [15037] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB98C6632679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15037] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15037] Picked up first job, doing one shot work
2016-03-13 03:11:35 GMT [VER] [15037] Processing secondary indexes
2016-03-13 03:11:35 GMT [VER] [15038] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15038] Starting backup for node BB9A81A2B679DD8
2016-03-13 03:11:35 GMT [VER] [15038] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15038] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9A81A2B679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15038] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15038] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15038] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9A81A2B679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15038] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15038] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15039] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15039] Starting backup for node BB908C52B679DD8
2016-03-13 03:11:35 GMT [VER] [15039] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15039] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB908C52B679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15039] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15039] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15039] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB908C52B679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15039] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15039] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15040] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15040] Starting backup for node BB90C6B32679DD8
2016-03-13 03:11:35 GMT [VER] [15040] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15040] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB90C6B32679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15040] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15040] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15040] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB90C6B32679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15040] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15040] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15041] Entering backup thread
2016-03-13 03:11:35 GMT [VER] [15040] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15041] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15041] Starting backup for node BB9107C32679DD8
2016-03-13 03:11:35 GMT [VER] [15041] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15041] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9107C32679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15041] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15041] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15041] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9107C32679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15041] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15041] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15042] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15042] Starting backup for node BB914C72A679DD8
2016-03-13 03:11:35 GMT [VER] [15042] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15042] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB914C72A679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15042] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15042] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15042] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB914C72A679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15042] Initializing backup file
7

PART 2:

2016-03-13 03:11:35 GMT [VER] [15042] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15043] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15043] Starting backup for node BB92C7C32679DD8
2016-03-13 03:11:35 GMT [VER] [15043] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15043] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB92C7C32679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15043] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15043] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15043] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB92C7C32679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15043] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15043] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15044] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15044] Starting backup for node BB938B832679DD8
2016-03-13 03:11:35 GMT [VER] [15044] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15044] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB938B832679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15044] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15044] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB938B832679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15044] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15044] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15044] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB938B832679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15044] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15044] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15045] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15045] Starting backup for node BB9443D32679DD8
2016-03-13 03:11:35 GMT [VER] [15045] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15045] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9443D32679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15045] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15045] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15045] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9443D32679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15045] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15045] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15046] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15046] Starting backup for node BB950B832679DD8
2016-03-13 03:11:35 GMT [VER] [15046] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15046] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB950B832679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15046] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15046] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15046] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB950B832679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15046] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15046] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15047] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15047] Starting backup for node BB958B832679DD8
2016-03-13 03:11:35 GMT [VER] [15047] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15047] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB958B832679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15047] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15047] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15047] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB958B832679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15047] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15047] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB958B832679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15047] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15047] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15048] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15048] Starting backup for node BB988B532679DD8
2016-03-13 03:11:35 GMT [VER] [15048] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15048] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB988B532679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15048] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15048] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15048] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB988B532679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15048] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15048] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15049] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15049] Starting backup for node BB990512B679DD8
2016-03-13 03:11:35 GMT [VER] [15049] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15049] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB990512B679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15049] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15049] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15049] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB990512B679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15049] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15049] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15036] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15036] Starting backup for node BB990862B679DD8
2016-03-13 03:11:35 GMT [VER] [15036] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15036] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB990862B679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15036] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15036] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15036] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB990862B679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15036] Initializing backup file
2016-03-13 03:11:35 GMT [VER] [15036] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [INF] [15037] No secondary indexes
2016-03-13 03:11:35 GMT [VER] [15036] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [INF] [15037] No secondary indexes
2016-03-13 03:11:35 GMT [VER] [15037] Processing UDFs
2016-03-13 03:11:35 GMT [VER] [15035] Entering backup thread
2016-03-13 03:11:35 GMT [INF] [15035] Starting backup for node BB9A0B432679DD8
2016-03-13 03:11:35 GMT [VER] [15035] 3293452449 remaining record(s), 0 B/rec average size
2016-03-13 03:11:35 GMT [VER] [15035] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9A0B432679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15035] Creating backup file
2016-03-13 03:11:35 GMT [VER] [15035] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 0 byte(s)
2016-03-13 03:11:35 GMT [INF] [15035] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9A0B432679DD8_00000.asb
2016-03-13 03:11:35 GMT [VER] [15035] Initializing backup file
2016-03-13 03:11:35 GMT [INF] [15037] Backing up 0 UDF file(s)
2016-03-13 03:11:35 GMT [VER] [15035] Ensuring one shot work completion
2016-03-13 03:11:35 GMT [VER] [15037] Signaling one shot work completion
2016-03-13 03:11:35 GMT [VER] [15034] Entering counter thread
2016-03-13 03:11:36 GMT [INF] [15034] 0% complete (~0 KiB/s, ~0 rec/s, ~0 B/rec)
2016-03-13 03:11:46 GMT [INF] [15034] 0% complete (~48352 KiB/s, ~287800 rec/s, ~172 B/rec)
2016-03-13 03:11:46 GMT [INF] [15034] ~3h10m32s remaining
2016-03-13 03:11:56 GMT [INF] [15034] 0% complete (~63047 KiB/s, ~375423 rec/s, ~171 B/rec)
2016-03-13 03:11:56 GMT [INF] [15034] ~2h25m55s remaining
2016-03-13 03:12:06 GMT [INF] [15034] 0% complete (~47135 KiB/s, ~280620 rec/s, ~172 B/rec)
2016-03-13 03:12:06 GMT [INF] [15034] ~3h15m2s remaining
2016-03-13 03:12:16 GMT [INF] [15034] 0% complete (~50989 KiB/s, ~303615 rec/s, ~171 B/rec)
2016-03-13 03:12:16 GMT [INF] [15034] ~3h0m5s remaining
2016-03-13 03:12:26 GMT [INF] [15034] 0% complete (~51683 KiB/s, ~307653 rec/s, ~172 B/rec)
2016-03-13 03:12:26 GMT [INF] [15034] ~2h57m32s remaining
2016-03-13 03:12:36 GMT [INF] [15034] 0% complete (~53366 KiB/s, ~317802 rec/s, ~171 B/rec)
2016-03-13 03:12:36 GMT [INF] [15034] ~2h51m42s remaining
2016-03-13 03:12:44 GMT [VER] [15035] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:44 GMT [VER] [15035] Closing backup file
2016-03-13 03:12:44 GMT [VER] [15035] Closing file descriptor
2016-03-13 03:12:44 GMT [VER] [15046] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:44 GMT [VER] [15035] Closing file descriptor
2016-03-13 03:12:44 GMT [VER] [15046] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:44 GMT [VER] [15046] Closing backup file
2016-03-13 03:12:44 GMT [VER] [15043] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:44 GMT [VER] [15043] Closing backup file
2016-03-13 03:12:44 GMT [VER] [15046] Closing file descriptor
2016-03-13 03:12:44 GMT [VER] [15043] Closing file descriptor
2016-03-13 03:12:45 GMT [VER] [15048] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:45 GMT [VER] [15048] Closing backup file
2016-03-13 03:12:45 GMT [VER] [15048] Closing file descriptor
2016-03-13 03:12:45 GMT [VER] [15047] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:45 GMT [VER] [15047] Closing backup file
2016-03-13 03:12:45 GMT [VER] [15047] Closing file descriptor
2016-03-13 03:12:45 GMT [VER] [15045] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:45 GMT [VER] [15045] Closing backup file
2016-03-13 03:12:45 GMT [VER] [15045] Closing file descriptor
2016-03-13 03:12:46 GMT [VER] [15040] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:46 GMT [VER] [15040] Closing backup file
2016-03-13 03:12:46 GMT [VER] [15040] Closing file descriptor
2016-03-13 03:12:46 GMT [VER] [15041] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:46 GMT [VER] [15041] Closing backup file
2016-03-13 03:12:46 GMT [VER] [15041] Closing file descriptor
2016-03-13 03:12:46 GMT [INF] [15034] 0% complete (~24573 KiB/s, ~146326 rec/s, ~171 B/rec)
2016-03-13 03:12:46 GMT [INF] [15034] ~6h12m34s remaining
2016-03-13 03:12:46 GMT [VER] [15044] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:46 GMT [VER] [15044] Closing backup file
2016-03-13 03:12:46 GMT [VER] [15044] Closing file descriptor
2016-03-13 03:12:47 GMT [VER] [15037] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:47 GMT [VER] [15037] Closing backup file
2016-03-13 03:12:48 GMT [VER] [15037] Closing file descriptor
2016-03-13 03:12:48 GMT [VER] [15038] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:48 GMT [VER] [15038] Closing backup file
2016-03-13 03:12:48 GMT [VER] [15038] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:48 GMT [VER] [15038] Closing backup file
2016-03-13 03:12:48 GMT [VER] [15038] Closing file descriptor
2016-03-13 03:12:49 GMT [VER] [15042] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:49 GMT [VER] [15042] Closing backup file
2016-03-13 03:12:49 GMT [VER] [15042] Closing file descriptor
2016-03-13 03:12:50 GMT [VER] [15036] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:50 GMT [VER] [15036] Closing backup file
2016-03-13 03:12:50 GMT [VER] [15036] Closing file descriptor
2016-03-13 03:12:51 GMT [VER] [15039] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:51 GMT [VER] [15039] Closing backup file
2016-03-13 03:12:51 GMT [VER] [15039] Closing file descriptor
2016-03-13 03:12:51 GMT [VER] [15049] Crossed 262144000 bytes, switching backup file
2016-03-13 03:12:51 GMT [VER] [15049] Closing backup file
2016-03-13 03:12:51 GMT [VER] [15049] Closing file descriptor
2016-03-13 03:12:56 GMT [INF] [15034] 0% complete (~0 KiB/s, ~0 rec/s, ~0 B/rec)
2016-03-13 03:13:02 GMT [VER] [15043] File size is 262144094
2016-03-13 03:13:02 GMT [VER] [15043] 3270591423 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15043] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB92C7C32679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15043] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15043] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562541724756 byte(s)
2016-03-13 03:13:02 GMT [INF] [15043] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB92C7C32679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15043] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15048] File size is 262144179
2016-03-13 03:13:02 GMT [VER] [15048] 3270591100 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15048] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB988B532679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15048] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15048] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562541669200 byte(s)
2016-03-13 03:13:02 GMT [INF] [15048] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB988B532679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15048] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15040] File size is 262144147
2016-03-13 03:13:02 GMT [VER] [15040] 3270590832 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15040] File size is 262144147
2016-03-13 03:13:02 GMT [VER] [15040] 3270590832 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15040] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB90C6B32679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15040] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15040] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562541623104 byte(s)
2016-03-13 03:13:02 GMT [INF] [15040] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB90C6B32679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15040] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15041] File size is 262144127
2016-03-13 03:13:02 GMT [VER] [15041] 3270589924 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15041] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9107C32679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15041] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15041] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562541466928 byte(s)
2016-03-13 03:13:02 GMT [INF] [15041] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9107C32679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15041] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15044] File size is 262144100
2016-03-13 03:13:02 GMT [VER] [15044] 3270589152 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15044] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB938B832679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15044] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15044] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562541334144 byte(s)
2016-03-13 03:13:02 GMT [INF] [15044] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB938B832679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15044] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15036] File size is 262144112
2016-03-13 03:13:02 GMT [VER] [15036] 3270588309 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15036] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB990862B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15036] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15036] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562541189148 byte(s)
2016-03-13 03:13:02 GMT [INF] [15036] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB990862B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15036] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15049] File size is 262144147
2016-03-13 03:13:02 GMT [VER] [15049] 3270587513 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15049] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB990512B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15049] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15049] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB990512B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15049] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15049] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562541052236 byte(s)
2016-03-13 03:13:02 GMT [INF] [15049] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB990512B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15049] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15038] File size is 262144132
2016-03-13 03:13:02 GMT [VER] [15038] 3270586678 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15038] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9A81A2B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15038] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15038] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562540908616 byte(s)
2016-03-13 03:13:02 GMT [INF] [15038] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9A81A2B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15038] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15047] File size is 262144256
2016-03-13 03:13:02 GMT [VER] [15047] 3270585908 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15047] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB958B832679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15047] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15047] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562540776176 byte(s)
2016-03-13 03:13:02 GMT [INF] [15047] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB958B832679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15047] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15046] File size is 262144019
2016-03-13 03:13:02 GMT [VER] [15046] 3270585636 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15046] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB950B832679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15046] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15046] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562540729392 byte(s)
2016-03-13 03:13:02 GMT [INF] [15046] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB950B832679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15046] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15039] File size is 262144068
2016-03-13 03:13:02 GMT [VER] [15039] 3270584820 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15039] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB908C52B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15039] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15039] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562540589040 byte(s)
2016-03-13 03:13:02 GMT [INF] [15039] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB908C52B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15039] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562540589040 byte(s)
2016-03-13 03:13:02 GMT [INF] [15039] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB908C52B679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15039] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15035] File size is 262144010
2016-03-13 03:13:02 GMT [VER] [15035] 3270584244 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15035] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9A0B432679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15035] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15035] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562540489968 byte(s)
2016-03-13 03:13:02 GMT [INF] [15035] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9A0B432679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15035] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15037] File size is 262144080
2016-03-13 03:13:02 GMT [VER] [15037] 3270583534 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15037] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB98C6632679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15037] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15037] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562540367848 byte(s)
2016-03-13 03:13:02 GMT [INF] [15037] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB98C6632679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15037] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15045] File size is 262144146
2016-03-13 03:13:02 GMT [VER] [15045] 3270582674 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15045] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9443D32679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15045] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15045] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562540219928 byte(s)
2016-03-13 03:13:02 GMT [INF] [15045] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9443D32679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15045] Initializing backup file
2016-03-13 03:13:02 GMT [VER] [15042] File size is 262144041
2016-03-13 03:13:02 GMT [VER] [15042] 3270581905 remaining record(s), 172 B/rec average size
2016-03-13 03:13:02 GMT [VER] [15042] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB914C72A679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15042] Creating backup file
2016-03-13 03:13:02 GMT [VER] [15042] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 562540087660 byte(s)
2016-03-13 03:13:02 GMT [INF] [15042] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB914C72A679DD8_00001.asb
2016-03-13 03:13:02 GMT [VER] [15042] Initializing backup file
2016-03-13 03:13:06 GMT [INF] [15034] 0% complete (~54308 KiB/s, ~323282 rec/s, ~172 B/rec)
2016-03-13 03:13:02 GMT [VER] [15042] Initializing backup file
2016-03-13 03:13:06 GMT [INF] [15034] 0% complete (~54308 KiB/s, ~323282 rec/s, ~172 B/rec)
2016-03-13 03:13:06 GMT [INF] [15034] ~2h48m31s remaining
2016-03-13 03:13:16 GMT [INF] [15034] 0% complete (~61935 KiB/s, ~368772 rec/s, ~171 B/rec)
2016-03-13 03:13:16 GMT [INF] [15034] ~2h27m35s remaining
2016-03-13 03:13:26 GMT [INF] [15034] 0% complete (~56829 KiB/s, ~338362 rec/s, ~171 B/rec)
2016-03-13 03:13:26 GMT [INF] [15034] ~2h40m41s remaining
2016-03-13 03:13:36 GMT [INF] [15034] 1% complete (~51176 KiB/s, ~304560 rec/s, ~172 B/rec)
2016-03-13 03:13:36 GMT [INF] [15034] ~2h58m20s remaining
2016-03-13 03:13:46 GMT [INF] [15034] 1% complete (~58132 KiB/s, ~346195 rec/s, ~171 B/rec)
2016-03-13 03:13:46 GMT [INF] [15034] ~2h36m44s remaining
2016-03-13 03:13:56 GMT [INF] [15034] 1% complete (~56337 KiB/s, ~335413 rec/s, ~171 B/rec)
2016-03-13 03:13:56 GMT [INF] [15034] ~2h41m37s remaining
2016-03-13 03:14:06 GMT [INF] [15034] 1% complete (~61324 KiB/s, ~364990 rec/s, ~172 B/rec)
2016-03-13 03:14:06 GMT [INF] [15034] ~2h28m22s remaining
2016-03-13 03:14:07 GMT [VER] [15035] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:07 GMT [VER] [15035] Closing backup file
2016-03-13 03:14:07 GMT [VER] [15035] Closing file descriptor
2016-03-13 03:14:08 GMT [VER] [15047] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:08 GMT [VER] [15047] Closing backup file
2016-03-13 03:14:08 GMT [VER] [15046] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:08 GMT [VER] [15046] Closing backup file
2016-03-13 03:14:08 GMT [VER] [15048] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:08 GMT [VER] [15048] Closing backup file
2016-03-13 03:14:08 GMT [VER] [15045] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:08 GMT [VER] [15045] Closing backup file
2016-03-13 03:14:09 GMT [VER] [15047] Closing file descriptor
2016-03-13 03:14:09 GMT [VER] [15048] Closing file descriptor
2016-03-13 03:14:09 GMT [VER] [15045] Closing file descriptor
2016-03-13 03:14:09 GMT [VER] [15046] Closing file descriptor
2016-03-13 03:14:09 GMT [VER] [15043] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:09 GMT [VER] [15043] Closing backup file
2016-03-13 03:14:09 GMT [VER] [15043] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:09 GMT [VER] [15043] Closing backup file
2016-03-13 03:14:09 GMT [VER] [15043] Closing file descriptor
2016-03-13 03:14:09 GMT [VER] [15040] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:09 GMT [VER] [15040] Closing backup file
2016-03-13 03:14:09 GMT [VER] [15040] Closing file descriptor
2016-03-13 03:14:10 GMT [VER] [15041] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:10 GMT [VER] [15041] Closing backup file
2016-03-13 03:14:10 GMT [VER] [15041] Closing file descriptor
2016-03-13 03:14:12 GMT [VER] [15037] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:12 GMT [VER] [15037] Closing backup file
2016-03-13 03:14:12 GMT [VER] [15037] Closing file descriptor
2016-03-13 03:14:12 GMT [VER] [15044] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:12 GMT [VER] [15044] Closing backup file
2016-03-13 03:14:12 GMT [VER] [15044] Closing file descriptor
2016-03-13 03:14:14 GMT [VER] [15042] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:14 GMT [VER] [15042] Closing backup file
2016-03-13 03:14:14 GMT [VER] [15042] Closing file descriptor
2016-03-13 03:14:14 GMT [VER] [15039] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:14 GMT [VER] [15039] Closing backup file
2016-03-13 03:14:14 GMT [VER] [15039] Closing file descriptor
2016-03-13 03:14:15 GMT [VER] [15038] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:15 GMT [VER] [15038] Closing backup file
2016-03-13 03:14:15 GMT [VER] [15038] Closing file descriptor
2016-03-13 03:14:15 GMT [VER] [15049] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:15 GMT [VER] [15049] Closing backup file
2016-03-13 03:14:15 GMT [VER] [15049] Closing file descriptor
2016-03-13 03:14:16 GMT [VER] [15036] Crossed 262144000 bytes, switching backup file
2016-03-13 03:14:16 GMT [VER] [15036] Closing backup file
2016-03-13 03:14:16 GMT [VER] [15036] Closing file descriptor
2016-03-13 03:14:16 GMT [INF] [15034] 1% complete (~3356 KiB/s, ~20033 rec/s, ~171 B/rec)
2016-03-13 03:14:16 GMT [INF] [15034] ~1d21h1m59s remaining
2016-03-13 03:14:16 GMT [INF] [15034] 1% complete (~3356 KiB/s, ~20033 rec/s, ~171 B/rec)
2016-03-13 03:14:16 GMT [INF] [15034] ~1d21h1m59s remaining
2016-03-13 03:14:23 GMT [VER] [15035] File size is 262144060
2016-03-13 03:14:23 GMT [VER] [15035] 3247730749 remaining record(s), 172 B/rec average size
8

PART 3:

2016-03-13 03:14:23 GMT [VER] [15035] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9A0B432679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15035] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15035] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558609688828 byte(s)
2016-03-13 03:14:23 GMT [INF] [15035] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9A0B432679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15035] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15037] File size is 262144084
2016-03-13 03:14:23 GMT [VER] [15037] 3247663639 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15037] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB98C6632679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15037] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15037] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558598145908 byte(s)
2016-03-13 03:14:23 GMT [INF] [15037] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB98C6632679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15037] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15047] File size is 262144147
2016-03-13 03:14:23 GMT [VER] [15047] 3247663221 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15047] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB958B832679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15047] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15047] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558598074012 byte(s)
2016-03-13 03:14:23 GMT [INF] [15047] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB958B832679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15047] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15043] File size is 262144061
2016-03-13 03:14:23 GMT [VER] [15043] 3247663024 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15043] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB92C7C32679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15043] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15043] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558598040128 byte(s)
2016-03-13 03:14:23 GMT [INF] [15043] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB92C7C32679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15043] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15045] File size is 262144070
2016-03-13 03:14:23 GMT [VER] [15045] 3247662566 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15045] File size is 262144070
2016-03-13 03:14:23 GMT [VER] [15045] 3247662566 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15045] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9443D32679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15045] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15045] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558597961352 byte(s)
2016-03-13 03:14:23 GMT [INF] [15045] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9443D32679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15045] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15041] File size is 262144059
2016-03-13 03:14:23 GMT [VER] [15041] 3247661930 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15041] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9107C32679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15041] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15041] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558597851960 byte(s)
2016-03-13 03:14:23 GMT [INF] [15041] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9107C32679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15041] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15046] File size is 262144123
2016-03-13 03:14:23 GMT [VER] [15046] 3247660987 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15046] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB950B832679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15046] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15046] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558597689764 byte(s)
2016-03-13 03:14:23 GMT [INF] [15046] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB950B832679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15046] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15044] File size is 262144136
2016-03-13 03:14:23 GMT [VER] [15044] 3247660289 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15044] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB938B832679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15044] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15044] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558597569708 byte(s)
2016-03-13 03:14:23 GMT [INF] [15044] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB938B832679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15044] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15040] File size is 262144157
2016-03-13 03:14:23 GMT [VER] [15040] 3247659326 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15040] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB90C6B32679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15040] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15040] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB90C6B32679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15040] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15040] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558597404072 byte(s)
2016-03-13 03:14:23 GMT [INF] [15040] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB90C6B32679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15040] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15042] File size is 262144036
2016-03-13 03:14:23 GMT [VER] [15042] 3247658403 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15042] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB914C72A679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15042] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15042] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558597245316 byte(s)
2016-03-13 03:14:23 GMT [INF] [15042] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB914C72A679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15042] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15039] File size is 262144060
2016-03-13 03:14:23 GMT [VER] [15039] 3247657573 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15039] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB908C52B679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15039] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15039] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558597102556 byte(s)
2016-03-13 03:14:23 GMT [INF] [15039] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB908C52B679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15039] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15036] File size is 262144017
2016-03-13 03:14:23 GMT [VER] [15036] 3247656614 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15036] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB990862B679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15036] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15036] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558596937608 byte(s)
2016-03-13 03:14:23 GMT [INF] [15036] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB990862B679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15036] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15038] File size is 262144161
2016-03-13 03:14:23 GMT [VER] [15038] 3247655794 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15038] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9A81A2B679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15038] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15038] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558596796568 byte(s)
2016-03-13 03:14:23 GMT [INF] [15038] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9A81A2B679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15038] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558596796568 byte(s)
2016-03-13 03:14:23 GMT [INF] [15038] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9A81A2B679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15038] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15049] File size is 262144120
2016-03-13 03:14:23 GMT [VER] [15049] 3247654970 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15049] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB990512B679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15049] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15049] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558596654840 byte(s)
2016-03-13 03:14:23 GMT [INF] [15049] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB990512B679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15049] Initializing backup file
2016-03-13 03:14:23 GMT [VER] [15048] File size is 262144143
2016-03-13 03:14:23 GMT [VER] [15048] 3247654137 remaining record(s), 172 B/rec average size
2016-03-13 03:14:23 GMT [VER] [15048] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB988B532679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15048] Creating backup file
2016-03-13 03:14:23 GMT [VER] [15048] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 558596511564 byte(s)
2016-03-13 03:14:23 GMT [INF] [15048] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB988B532679DD8_00002.asb
2016-03-13 03:14:23 GMT [VER] [15048] Initializing backup file
2016-03-13 03:14:26 GMT [INF] [15034] 1% complete (~66826 KiB/s, ~397834 rec/s, ~172 B/rec)
2016-03-13 03:14:26 GMT [INF] [15034] ~2h16m0s remaining
2016-03-13 03:14:36 GMT [INF] [15034] 1% complete (~46603 KiB/s, ~277406 rec/s, ~172 B/rec)
2016-03-13 03:14:36 GMT [INF] [15034] ~3h14m51s remaining
2016-03-13 03:14:46 GMT [INF] [15034] 1% complete (~55242 KiB/s, ~328816 rec/s, ~172 B/rec)
2016-03-13 03:14:46 GMT [INF] [15034] ~2h44m13s remaining
2016-03-13 03:14:46 GMT [VER] [15049] Received scan end marker
2016-03-13 03:14:46 GMT [INF] [15049] Completed backup for node BB990512B679DD8, records: 3565014, size: 611525685 (~171 B/rec)
2016-03-13 03:14:46 GMT [VER] [15049] Closing backup file
2016-03-13 03:14:46 GMT [VER] [15049] Closing file descriptor
2016-03-13 03:14:47 GMT [VER] [15049] File size is 87237418
2016-03-13 03:14:47 GMT [VER] [15049] Job queue is empty
2016-03-13 03:14:47 GMT [VER] [15049] Leaving backup thread
2016-03-13 03:14:56 GMT [INF] [15034] 1% complete (~47913 KiB/s, ~285208 rec/s, ~172 B/rec)
2016-03-13 03:14:56 GMT [INF] [15034] ~3h9m9s remaining
2016-03-13 03:14:56 GMT [INF] [15034] 1% complete (~47913 KiB/s, ~285208 rec/s, ~172 B/rec)
2016-03-13 03:14:56 GMT [INF] [15034] ~3h9m9s remaining
2016-03-13 03:15:05 GMT [VER] [15040] Received scan end marker
2016-03-13 03:15:05 GMT [INF] [15040] Completed backup for node BB90C6B32679DD8, records: 3967684, size: 683426416 (~172 B/rec)
2016-03-13 03:15:05 GMT [VER] [15040] Closing backup file
2016-03-13 03:15:05 GMT [VER] [15040] Closing file descriptor
2016-03-13 03:15:06 GMT [INF] [15034] 1% complete (~55025 KiB/s, ~327478 rec/s, ~172 B/rec)
2016-03-13 03:15:06 GMT [INF] [15034] ~2h44m34s remaining
2016-03-13 03:15:09 GMT [VER] [15041] Received scan end marker
2016-03-13 03:15:09 GMT [INF] [15041] Completed backup for node BB9107C32679DD8, records: 4125923, size: 709309009 (~171 B/rec)
2016-03-13 03:15:09 GMT [VER] [15041] Closing backup file
2016-03-13 03:15:09 GMT [VER] [15041] Closing file descriptor
2016-03-13 03:15:10 GMT [VER] [15040] File size is 159138112
2016-03-13 03:15:10 GMT [VER] [15040] Job queue is empty
2016-03-13 03:15:10 GMT [VER] [15040] Leaving backup thread
2016-03-13 03:15:12 GMT [VER] [15041] File size is 185020823
2016-03-13 03:15:12 GMT [VER] [15041] Job queue is empty
2016-03-13 03:15:12 GMT [VER] [15041] Leaving backup thread
2016-03-13 03:15:13 GMT [VER] [15045] Received scan end marker
2016-03-13 03:15:13 GMT [INF] [15045] Completed backup for node BB9443D32679DD8, records: 4172688, size: 717990331 (~172 B/rec)
2016-03-13 03:15:13 GMT [VER] [15045] Closing backup file
2016-03-13 03:15:13 GMT [VER] [15045] Closing file descriptor
2016-03-13 03:15:16 GMT [INF] [15034] 1% complete (~36998 KiB/s, ~220334 rec/s, ~171 B/rec)
2016-03-13 03:15:16 GMT [INF] [15034] ~4h4m25s remaining
2016-03-13 03:15:17 GMT [VER] [15042] Received scan end marker
2016-03-13 03:15:17 GMT [INF] [15042] Completed backup for node BB914C72A679DD8, records: 4160742, size: 715936998 (~172 B/rec)
2016-03-13 03:15:17 GMT [VER] [15042] Closing backup file
2016-03-13 03:15:17 GMT [VER] [15042] Closing file descriptor
2016-03-13 03:15:20 GMT [VER] [15045] File size is 193702115
2016-03-13 03:15:20 GMT [VER] [15045] Job queue is empty
2016-03-13 03:15:20 GMT [VER] [15045] Leaving backup thread
2016-03-13 03:15:22 GMT [VER] [15042] File size is 191648921
2016-03-13 03:15:20 GMT [VER] [15045] Leaving backup thread
2016-03-13 03:15:22 GMT [VER] [15042] File size is 191648921
2016-03-13 03:15:22 GMT [VER] [15042] Job queue is empty
2016-03-13 03:15:22 GMT [VER] [15042] Leaving backup thread
2016-03-13 03:15:22 GMT [VER] [15048] Received scan end marker
2016-03-13 03:15:22 GMT [INF] [15048] Completed backup for node BB988B532679DD8, records: 4408120, size: 758912556 (~172 B/rec)
2016-03-13 03:15:22 GMT [VER] [15048] Closing backup file
2016-03-13 03:15:22 GMT [VER] [15048] Closing file descriptor
2016-03-13 03:15:23 GMT [VER] [15039] Received scan end marker
2016-03-13 03:15:23 GMT [INF] [15039] Completed backup for node BB908C52B679DD8, records: 4320727, size: 739348747 (~171 B/rec)
2016-03-13 03:15:23 GMT [VER] [15039] Closing backup file
2016-03-13 03:15:23 GMT [VER] [15039] Closing file descriptor
2016-03-13 03:15:26 GMT [INF] [15034] 1% complete (~25734 KiB/s, ~153022 rec/s, ~172 B/rec)
2016-03-13 03:15:26 GMT [INF] [15034] ~5h51m43s remaining
2016-03-13 03:15:29 GMT [VER] [15046] Crossed 262144000 bytes, switching backup file
2016-03-13 03:15:29 GMT [VER] [15046] Closing backup file
2016-03-13 03:15:29 GMT [VER] [15046] Closing file descriptor
2016-03-13 03:15:29 GMT [VER] [15047] Crossed 262144000 bytes, switching backup file
2016-03-13 03:15:29 GMT [VER] [15047] Closing backup file
2016-03-13 03:15:29 GMT [VER] [15047] Closing file descriptor
2016-03-13 03:15:29 GMT [VER] [15039] File size is 215060619
2016-03-13 03:15:29 GMT [VER] [15039] Job queue is empty
2016-03-13 03:15:29 GMT [VER] [15039] Leaving backup thread
2016-03-13 03:15:30 GMT [VER] [15036] Received scan end marker
2016-03-13 03:15:30 GMT [INF] [15036] Completed backup for node BB990862B679DD8, records: 4396434, size: 756457880 (~172 B/rec)
2016-03-13 03:15:30 GMT [VER] [15036] Closing backup file
2016-03-13 03:15:30 GMT [VER] [15035] Crossed 262144000 bytes, switching backup file
2016-03-13 03:15:30 GMT [VER] [15035] Closing backup file
2016-03-13 03:15:31 GMT [VER] [15044] Received scan end marker
2016-03-13 03:15:31 GMT [INF] [15044] Completed backup for node BB938B832679DD8, records: 4541407, size: 782003349 (~172 B/rec)
2016-03-13 03:15:31 GMT [VER] [15044] Closing backup file
2016-03-13 03:15:31 GMT [VER] [15043] Received scan end marker
2016-03-13 03:15:31 GMT [VER] [15044] Closing backup file
2016-03-13 03:15:31 GMT [VER] [15043] Received scan end marker
2016-03-13 03:15:31 GMT [INF] [15043] Completed backup for node BB92C7C32679DD8, records: 4558872, size: 783552425 (~171 B/rec)
2016-03-13 03:15:31 GMT [VER] [15043] Closing backup file
2016-03-13 03:15:31 GMT [VER] [15044] Closing file descriptor
2016-03-13 03:15:31 GMT [VER] [15043] Closing file descriptor
2016-03-13 03:15:31 GMT [VER] [15035] Closing file descriptor
2016-03-13 03:15:31 GMT [VER] [15036] Closing file descriptor
2016-03-13 03:15:33 GMT [VER] [15037] Crossed 262144000 bytes, switching backup file
2016-03-13 03:15:33 GMT [VER] [15037] Closing backup file
2016-03-13 03:15:33 GMT [VER] [15037] Closing file descriptor
2016-03-13 03:15:36 GMT [INF] [15034] 1% complete (~398 KiB/s, ~2375 rec/s, ~171 B/rec)
2016-03-13 03:15:36 GMT [INF] [15034] ~15d17h35m45s remaining
2016-03-13 03:15:37 GMT [VER] [15038] Crossed 262144000 bytes, switching backup file
2016-03-13 03:15:37 GMT [VER] [15038] Closing backup file
2016-03-13 03:15:37 GMT [VER] [15038] Closing file descriptor
2016-03-13 03:15:46 GMT [INF] [15034] 1% complete (~0 KiB/s, ~0 rec/s, ~0 B/rec)
2016-03-13 03:15:48 GMT [VER] [15048] File size is 234624234
2016-03-13 03:15:48 GMT [VER] [15048] Job queue is empty
2016-03-13 03:15:48 GMT [VER] [15048] Leaving backup thread
2016-03-13 03:15:48 GMT [VER] [15046] File size is 262144014
2016-03-13 03:15:48 GMT [VER] [15046] 3228395111 remaining record(s), 172 B/rec average size
2016-03-13 03:15:48 GMT [VER] [15046] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB950B832679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15046] Creating backup file
2016-03-13 03:15:48 GMT [VER] [15046] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 555283959092 byte(s)
2016-03-13 03:15:48 GMT [INF] [15046] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB950B832679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15046] Initializing backup file
2016-03-13 03:15:48 GMT [VER] [15047] File size is 262144117
2016-03-13 03:15:48 GMT [VER] [15047] 3228394640 remaining record(s), 172 B/rec average size
2016-03-13 03:15:48 GMT [VER] [15047] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB958B832679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15047] Creating backup file
2016-03-13 03:15:48 GMT [VER] [15047] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 555283878080 byte(s)
2016-03-13 03:15:48 GMT [VER] [15047] Creating backup file
2016-03-13 03:15:48 GMT [VER] [15047] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 555283878080 byte(s)
2016-03-13 03:15:48 GMT [INF] [15047] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB958B832679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15047] Initializing backup file
2016-03-13 03:15:48 GMT [VER] [15043] File size is 259264270
2016-03-13 03:15:48 GMT [VER] [15043] Job queue is empty
2016-03-13 03:15:48 GMT [VER] [15043] Leaving backup thread
2016-03-13 03:15:48 GMT [VER] [15044] File size is 257715113
2016-03-13 03:15:48 GMT [VER] [15044] Job queue is empty
2016-03-13 03:15:48 GMT [VER] [15044] Leaving backup thread
2016-03-13 03:15:48 GMT [VER] [15035] File size is 262144015
2016-03-13 03:15:48 GMT [VER] [15035] 3228393868 remaining record(s), 172 B/rec average size
2016-03-13 03:15:48 GMT [VER] [15035] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9A0B432679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15035] Creating backup file
2016-03-13 03:15:48 GMT [VER] [15035] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 555283745296 byte(s)
2016-03-13 03:15:48 GMT [INF] [15035] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9A0B432679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15035] Initializing backup file
2016-03-13 03:15:48 GMT [VER] [15037] File size is 262144112
2016-03-13 03:15:48 GMT [VER] [15037] 3228393315 remaining record(s), 172 B/rec average size
2016-03-13 03:15:48 GMT [VER] [15037] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB98C6632679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15037] Creating backup file
2016-03-13 03:15:48 GMT [VER] [15037] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 555283650180 byte(s)
2016-03-13 03:15:48 GMT [INF] [15037] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB98C6632679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15037] Initializing backup file
2016-03-13 03:15:48 GMT [VER] [15038] File size is 262144083
2016-03-13 03:15:48 GMT [VER] [15038] 3228392613 remaining record(s), 172 B/rec average size
2016-03-13 03:15:48 GMT [VER] [15038] Opening backup file /data/aerospike/sets/SnapshotHashes/all/BB9A81A2B679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15038] Creating backup file
2016-03-13 03:15:48 GMT [VER] [15038] Checking disk space on /data/aerospike/sets/SnapshotHashes/all for 555283529436 byte(s)
2016-03-13 03:15:48 GMT [INF] [15038] Created new backup file /data/aerospike/sets/SnapshotHashes/all/BB9A81A2B679DD8_00003.asb
2016-03-13 03:15:48 GMT [VER] [15038] Initializing backup file
2016-03-13 03:15:48 GMT [VER] [15036] File size is 232169751
2016-03-13 03:15:48 GMT [VER] [15038] Initializing backup file
2016-03-13 03:15:48 GMT [VER] [15036] File size is 232169751
2016-03-13 03:15:48 GMT [VER] [15036] Job queue is empty
2016-03-13 03:15:48 GMT [VER] [15036] Leaving backup thread
2016-03-13 03:15:51 GMT [VER] [15046] Received scan end marker
2016-03-13 03:15:51 GMT [INF] [15046] Completed backup for node BB950B832679DD8, records: 4652474, size: 801025504 (~172 B/rec)
2016-03-13 03:15:51 GMT [VER] [15046] Closing backup file
2016-03-13 03:15:51 GMT [VER] [15046] Closing file descriptor
2016-03-13 03:15:51 GMT [VER] [15046] File size is 14593348
2016-03-13 03:15:51 GMT [VER] [15046] Job queue is empty
2016-03-13 03:15:51 GMT [VER] [15046] Leaving backup thread
2016-03-13 03:15:56 GMT [INF] [15034] 2% complete (~17172 KiB/s, ~102128 rec/s, ~172 B/rec)
2016-03-13 03:15:56 GMT [INF] [15034] ~8h46m41s remaining
2016-03-13 03:15:57 GMT [VER] [15037] Received scan end marker
2016-03-13 03:15:57 GMT [INF] [15037] Completed backup for node BB98C6632679DD8, records: 4799163, size: 826459872 (~172 B/rec)
2016-03-13 03:15:57 GMT [VER] [15037] Closing backup file
2016-03-13 03:15:57 GMT [VER] [15037] Closing file descriptor
2016-03-13 03:15:58 GMT [VER] [15037] File size is 40027596
2016-03-13 03:15:58 GMT [VER] [15037] Job queue is empty
2016-03-13 03:15:58 GMT [VER] [15037] Leaving backup thread
2016-03-13 03:16:01 GMT [VER] [15038] Received scan end marker
2016-03-13 03:16:01 GMT [INF] [15038] Completed backup for node BB9A81A2B679DD8, records: 4858249, size: 836355611 (~172 B/rec)
2016-03-13 03:16:01 GMT [VER] [15038] Closing backup file
2016-03-13 03:16:02 GMT [VER] [15038] Closing file descriptor
2016-03-13 03:16:02 GMT [VER] [15038] File size is 49923235
2016-03-13 03:16:02 GMT [VER] [15038] Job queue is empty
2016-03-13 03:16:02 GMT [VER] [15038] Leaving backup thread
2016-03-13 03:16:06 GMT [INF] [15034] 2% complete (~9151 KiB/s, ~54422 rec/s, ~172 B/rec)
2016-03-13 03:16:06 GMT [INF] [15034] ~16h28m13s remaining
2016-03-13 03:16:08 GMT [VER] [15035] Received scan end marker
2016-03-13 03:16:08 GMT [INF] [15035] Completed backup for node BB9A0B432679DD8, records: 5052948, size: 869164207 (~172 B/rec)
2016-03-13 03:16:08 GMT [VER] [15035] Closing backup file
2016-03-13 03:16:08 GMT [INF] [15035] Completed backup for node BB9A0B432679DD8, records: 5052948, size: 869164207 (~172 B/rec)
2016-03-13 03:16:08 GMT [VER] [15035] Closing backup file
2016-03-13 03:16:08 GMT [VER] [15035] Closing file descriptor
2016-03-13 03:16:09 GMT [VER] [15035] File size is 82732122
2016-03-13 03:16:09 GMT [VER] [15035] Job queue is empty
2016-03-13 03:16:09 GMT [VER] [15035] Leaving backup thread
2016-03-13 03:16:12 GMT [VER] [15047] Received scan end marker
2016-03-13 03:16:12 GMT [INF] [15047] Completed backup for node BB958B832679DD8, records: 5138934, size: 885293570 (~172 B/rec)
2016-03-13 03:16:12 GMT [VER] [15047] Closing backup file
2016-03-13 03:16:12 GMT [VER] [15047] Closing file descriptor
2016-03-13 03:16:13 GMT [VER] [15047] File size is 98861050
2016-03-13 03:16:13 GMT [VER] [15047] Job queue is empty
2016-03-13 03:16:13 GMT [VER] [15047] Leaving backup thread
2016-03-13 03:16:13 GMT [VER] [15016] Waiting for counter thread
2016-03-13 03:16:13 GMT [INF] [15034] Backed up 66719379 record(s), 0 secondary index(es), 0 UDF file(s) from 15 node(s), 11476762160 byte(s) in total (~172 B/rec)
2016-03-13 03:16:13 GMT [VER] [15034] Leaving counter thread
2016-03-13 03:16:13 GMT [VER] [15016] Exiting with status code 0
9

As you can see in PART 3, last percentage reported was 2% and then suddenly it finished!

AMC showes 135,970,769 objects exists in that Set but asbackup shows 66,719,379 records backed up! What may caused such a difference?

10

The set statistics include primary and replica counts. In the backup, only the primary copy is written.