# Urgent: Migration stuck v3.8.1, missing acks from node **URL:** \ **Category:** General Discussion\ **Tags:** migration\ **Created:** [June 20, 2017, 2:26pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289 "2017-06-20T14:26:47Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 20, 2017, 2:26pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/1 "2017-06-20T14:26:47Z")
I have 8 node cluster on 3.8.1 version. I have restarted one node because client connection was ~15k on this machine (on other machine ~3-5k) and migration started as expected but it got stuck on all machines after certain %. I increased migration thread to 5. ` asinfo -v "set-config:context=service;migrate-threads=5"` and it started again but got stuck again and i increased it to 10 then 15. Migration got completed on 1 node but got stuck to ~98-99% on all other nodes. When i checked logs(/var/log/aerospike/aerospike.log), it says - ``` WARNING (migrate): (migrate.c:930) missing acks from node bb9a4083a7ac40c ``` on some machines. when i checked on node bb9a4083a7ac40c (same machine that i restarted) it has repeated warning in logs: ``` Jun 20 2017 14:19:47 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb904d0657ac40c Jun 20 2017 14:19:49 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb904d0657ac40c Jun 20 2017 14:19:49 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb97cd1657ac40c Jun 20 2017 14:19:49 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb9f0d0657ac40c Jun 20 2017 14:19:49 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb9f0d0657ac40c Jun 20 2017 14:19:50 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb99c083a7ac40c Jun 20 2017 14:19:52 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb9f0d0657ac40c Jun 20 2017 14:19:53 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb904d0657ac40c Jun 20 2017 14:19:54 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb904d0657ac40c Jun 20 2017 14:19:54 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb97cd1657ac40c Jun 20 2017 14:19:54 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb9f0d0657ac40c Jun 20 2017 14:19:55 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb9f0d0657ac40c Jun 20 2017 14:19:55 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb99c083a7ac40c Jun 20 2017 14:19:57 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb9f0d0657ac40c ``` **My asmonitor info output:** ``` asmonitor -e info Enter help for commands Creating History file, [Errno 2] No such file or directory Cannot read config file /home/platform/.asmonitor/asmonitor.conf Created initial config file /home/platform/.asmonitor/asmonitor.conf 8 hosts in cluster: 10.100.156.40:3000,10.56.101.152:3000,10.56.101.153:3000,10.56.101.158:3000,10.56.101.184:3000,10.57.115.42:3000,10.57.115.79:3000,10.57.115.91:3000 ===NODES=== 2017-06-20 14:03:23.781525 Sorting by IP, in Ascending order: ip:port Build Cluster Cluster Free Free Migrates Node Principal Replicated Sys . Size Visibility Disk Mem . ID ID Objects Free . . . pct pct . . . . Mem 10.100.156.40:3000 3.8.1 8 true 93 28 (1,0) BB97CC4FA902500 BB9F0D0657AC40C 255,568,473 21 10.56.101.152:3000 3.8.1 8 true 93 30 (1,1) BB97CD1657AC40C BB9F0D0657AC40C 250,739,669 21 10.56.101.153:3000 3.8.1 8 true 93 30 (1,1) BB99C083A7AC40C BB9F0D0657AC40C 249,529,073 21 10.56.101.158:3000 3.8.1 8 true 93 31 (0,3) BB9F0D0657AC40C BB9F0D0657AC40C 246,399,098 23 10.56.101.184:3000 3.8.1 8 true 93 31 (0,0) BB9E047FB902500 BB9F0D0657AC40C 247,875,947 23 10.57.115.42:3000 3.8.1 8 true 93 33 (7,10) BB9A4083A7AC40C BB9F0D0657AC40C 242,836,781 25 10.57.115.79:3000 3.8.1 8 true 93 29 (0,0) BB9E0D0657AC40C BB9F0D0657AC40C 252,761,992 20 10.57.115.91:3000 3.8.1 8 true 93 28 (1,2) BB904D0657AC40C BB9F0D0657AC40C 255,422,182 20 Number of nodes displayed: 8 ===NAMESPACE=== Total (unique) objects in cluster for test : 0 Total (unique) objects in cluster for bidder : 1,000,971,169 Note: Total (unique) objects is an under estimate if migrations are in progress. ip/namespace Avail Evicted Master Repl Stop Used Used Used Used hwm hwm Pct Objects Objects Factor Writes Disk Disk Mem Mem Disk Mem . . . . . . % . % . . 10.56.101.184/bidder 91 0 124,286,992 2 false 62.88 G 7 41.37 G 83 50 60 10.56.101.152/bidder 91 0 131,351,331 2 false 63.60 G 7 41.87 G 84 50 60 10.57.115.91/bidder 91 0 129,163,656 2 false 64.79 G 7 42.66 G 86 50 60 10.56.101.158/bidder 91 0 123,056,943 2 false 62.51 G 7 41.14 G 83 50 60 10.57.115.79/bidder 91 0 124,544,372 2 false 64.12 G 7 42.17 G 85 50 60 10.56.101.153/bidder 91 0 120,054,404 2 false 63.30 G 7 41.59 G 84 50 60 10.100.156.40/bidder 91 0 132,088,662 2 false 64.83 G 7 42.64 G 86 50 60 10.57.115.42/bidder 93 0 116,424,809 2 false 61.60 G 7 39.61 G 80 50 60 10.56.101.184/test n/a 0 0 2 false n/a n/a 0.00 B 0 50 60 10.56.101.152/test n/a 0 0 2 false n/a n/a 0.00 B 0 50 60 10.57.115.91/test n/a 0 0 2 false n/a n/a 0.00 B 0 50 60 10.56.101.158/test n/a 0 0 2 false n/a n/a 0.00 B 0 50 60 10.57.115.79/test n/a 0 0 2 false n/a n/a 0.00 B 0 50 60 10.56.101.153/test n/a 0 0 2 false n/a n/a 0.00 B 0 50 60 10.57.115.42/test n/a 0 0 2 false n/a n/a 0.00 B 0 50 60 10.100.156.40/test n/a 0 0 2 false n/a n/a 0.00 B 0 50 60 Number of rows displayed: 16 ``` what’s going wrong ? ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [June 20, 2017, 6:59pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/2 "2017-06-20T18:59:54Z")
> [@nishant.nitjsr](#): > > Jun 20 2017 14:19:54 GMT: WARNING (migrate): (migrate.c:930) missing acks from node bb904d0657ac40c This means that a particular migration thread has begun to throttle because it has 16 MB of un-acked records. The warning doesn’t print until the thread has remained in this state for at least 5 seconds, and will continue to print in 5 second intervals until the outstanding acks drop below the 16 MB threshold. In other words, you have pushed migration higher than your network or machine/disks can sustain. I suggest decreasing migrate-threads down to 1 (this will not take effect immediately, threads will terminate as migrations complete). You should also increasing [migrate-sleep](http://www.aerospike.com/docs/reference/configuration#migrate-sleep) to ~200 μs. You may revert the migrate-sleep to the default value once the system stabilizes again. By default, migrate-sleep is 1 μs, which is aggressive but works well with one thread. With multiple threads this default can quickly become too aggressive. If you plan to tune up threads, begin by increasing the migrate-sleep to avoid overwhelming network etc. In later releases we introduced log tickers and stats to show the rate migrations are receiving and sending data which can be useful to see if you have tuned up the rate or not. ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 21, 2017, 3:17am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/3 "2017-06-21T03:17:48Z")
i have decreased migrate-thread to 1 and increased migrate-sleep to 200 ``` asinfo -v "set-config:context=namespace;id=bidder;migrate-sleep=200" asinfo -v "set-config:context=service;migrate-threads=1" ``` How long will it take to affect (approx) (I have \> 1B keys in cluster) ? will it be safe to add more nodes at this moment or should i wait for this migration to complete? I am running out of space so have to add new machines ASAP. Please suggest. ---
**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\ **Post date:** [June 21, 2017, 3:19am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/4 "2017-06-21T03:19:09Z")
You can add more machines now without issue, but make sure you wait at least 10 seconds between adding each node or until the cluster size is agreed on all the nodes. ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [June 21, 2017, 5:39am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/5 "2017-06-21T05:39:49Z")
Actually, adding a node would be great because it will cancel the ongoing migrations and your migrate threads will quickly reduce to the configured value. ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 21, 2017, 7:25am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/6 "2017-06-21T07:25:36Z")
Thanks it worked 😅 I added 2 more node and migration completed on all nodes. I also changed migrate-sleep on all nodes ``` asinfo -v "set-config:context=namespace;id=bidder;migrate-sleep=1" ``` ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 23, 2017, 9:00am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/7 "2017-06-23T09:00:05Z")
It got stuck again on migration ☹ . It seems that one node (BB9A4083A7AC40C) restarted during night but don’t know the reason. I caught few suspicious lines in log and posting it below: **asadm -e info** ``` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node Node Ip Build Enterprise Cluster Cluster Cluster Principal Client Uptime . Id . . . Size Key Integrity . Conns . 10.100.15.18:3000 BB95CD5FA902500 10.100.15.18:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 4287 52:58:57 10.100.15.96:3000 BB9E88A637AC40C 10.100.15.96:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 4208 52:30:11 10.100.156.40:3000 BB97CC4FA902500 10.100.156.40:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 4142 2090:33:50 10.56.101.152:3000 BB97CD1657AC40C 10.56.101.152:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 4099 2038:42:25 10.56.101.153:3000 BB99C083A7AC40C 10.56.101.153:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 4220 2038:11:58 10.56.101.158:3000 *BB9F0D0657AC40C 10.56.101.158:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 4099 2087:49:29 10.56.101.184:3000 BB9E047FB902500 10.56.101.184:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 4175 2037:41:37 10.57.115.42:3000 BB9A4083A7AC40C 10.57.115.42:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 6183 70:35:55 10.57.115.79:3000 BB9E0D0657AC40C 10.57.115.79:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 4108 2039:45:59 10.57.115.91:3000 BB904D0657AC40C 10.57.115.91:3000 3.8.1 False 10 1F7BC27D9A3EA972 True BB9F0D0657AC40C 4151 2085:24:18 Number of rows: 10 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Namespace Node Avail% Evictions Master Replica Repl Stop Pending Disk Disk HWM Mem Mem HWM Stop . . . . Objects Objects Factor Writes Migrates Used Used% Disk% Used Used% Mem% Writes% . . . . . . . . (tx%,rx%) . . . . . . . bidder 10.100.15.18:3000 92 0 110.569 M 112.270 M 2 false (45,100) 59.440 GB 6 50 38.159 GB 77 60 90 bidder 10.100.15.96:3000 93 0 114.565 M 98.603 M 2 false (48,100) 56.713 GB 6 50 36.409 GB 73 60 90 bidder 10.56.101.184:3000 91 0 118.758 M 103.662 M 2 false (50,100) 58.686 GB 6 50 38.604 GB 78 60 90 bidder 10.56.101.152:3000 91 0 122.076 M 95.552 M 2 false (41,100) 58.412 GB 6 50 38.475 GB 77 60 90 bidder 10.57.115.91:3000 91 0 124.452 M 107.362 M 2 false (62,100) 61.200 GB 7 50 40.281 GB 81 60 90 bidder 10.56.101.158:3000 91 0 114.767 M 103.964 M 2 false (54,100) 57.972 GB 6 50 38.164 GB 77 60 90 bidder 10.57.115.79:3000 91 0 117.346 M 106.236 M 2 false (55,100) 59.125 GB 6 50 38.890 GB 78 60 90 bidder 10.56.101.153:3000 91 0 112.623 M 108.844 M 2 false (48,98) 58.947 GB 6 50 38.733 GB 78 60 90 bidder 10.57.115.42:3000 92 0 111.221 M 283.523 K 2 false (99,50) 57.735 GB 6 50 37.066 GB 75 60 90 bidder 10.100.156.40:3000 91 0 120.929 M 104.353 M 2 false (47,100) 61.207 GB 7 50 40.241 GB 81 60 90 bidder 0 1.167 G 941.130 M (75,75) 589.436 GB 385.023 GB test 10.100.15.18:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 10.100.15.96:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 10.56.101.184:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 10.56.101.152:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 10.57.115.91:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 10.56.101.158:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 10.57.115.79:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 10.56.101.153:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 10.57.115.42:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 10.100.156.40:3000 N/E 0 0.000 0.000 2 false (0,0) N/E N/E 50 0.000 B 0 60 90 test 0 0.000 0.000 (0,0) 0.000 B 0.000 B Number of rows: 22 ``` Node BB9F0D0657AC40C logs: ``` Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2352) hb considers expiring: now 7996910434 last 7996908887 Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2371) hb expires but fabric says DEAD: node bb9a4083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2550) removing node on heartbeat failure: bb9a4083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (fabric): (fabric.c:1762) fabric: node bb9a4083a7ac40c departed Jun 22 2017 23:14:20 GMT: INFO (fabric): (fabric.c:1678) fabric disconnecting node: bb9a4083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1964) as_paxos_spark establishing transaction [33]@bb9f0d0657ac40c ClSz = 10 ; # change = 1 : DEL bb9a4083a7ac40c; Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:2932) {33} principal acking it's prepare bb9f0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1623) removing failed node bb9a4083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:356) DISALLOW MIGRATIONS Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb9e88a637ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb9e0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb9e047fb902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb99c083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb97cd1657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb97cc4fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb95cd5fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1509) SUCCESSION [33]@bb9f0d0657ac40c: bb9f0d0657ac40c bb9e88a637ac40c bb9e0d0657ac40c bb9e047fb902500 bb99c083a7ac40c bb97cd1657ac40c bb97cc4fa902500 bb95cd5fa902500 bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb99c083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb97cd1657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb9e047fb902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb97cc4fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb9e0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb9e88a637ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb95cd5fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3228) All partition data has been received by principal Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb9e88a637ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb9e0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb9e047fb902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb99c083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb97cd1657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb97cc4fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb95cd5fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2447) CLUSTER SIZE = 9 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2485) Global state is well formed Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2230) setting replication factors: cluster size 9, paxos single replica limit 1 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2245) {test} replication factor is 2 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2245) {bidder} replication factor is 2 Jun 22 2017 23:14:20 GMT: INFO (config): (cluster_config.c:421) rack aware is disabled Jun 22 2017 23:14:20 GMT: INFO (partition): (cluster_config.c:380) rack aware is disabled Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:3222) {test} re-balanced, expected migrations - (86 tx, 109 rx) Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:3222) {bidder} re-balanced, expected migrations - (86 tx, 109 rx) Jun 22 2017 23:14:20 GMT: INFO (paxos): (partition.c:3239) global partition state: total 8192 lost 0 unique 8192 duplicate 0 Jun 22 2017 23:14:20 GMT: INFO (paxos): (partition.c:3241) partition state after fixing lost partitions (master): total 8192 lost 0 unique 8192 duplicate 0 Jun 22 2017 23:14:20 GMT: INFO (paxos): (partition.c:3243) 1588 new partition version tree paths generated Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:348) ALLOW MIGRATIONS Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:1070) initiated connection to mesh seed host at 10.57.115.42:3002 (10.57.115.42:3002) via socket 393 from 10.56.101.158:56612 Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2556) new heartbeat received: bb9a4083a7ac40c principal node is bb9f0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (fabric): (fabric.c:1756) fabric: node bb9a4083a7ac40c arrived Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1964) as_paxos_spark establishing transaction [34]@bb9f0d0657ac40c ClSz = 9 ; # change = 1 : ADD bb9a4083a7ac40c; Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:2932) {34} principal acking it's prepare bb9f0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1616) inserting node bb9a4083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:356) DISALLOW MIGRATIONS Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb9e88a637ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb9e0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb9e047fb902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb9a4083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb99c083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb97cd1657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb97cc4fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb95cd5fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1500) sending sync message to bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:1509) SUCCESSION [34]@bb9f0d0657ac40c: bb9f0d0657ac40c bb9e88a637ac40c bb9e0d0657ac40c bb9e047fb902500 bb9a4083a7ac40c bb99c083a7ac40c bb97cd1657ac40c bb97cc4fa902500 bb95cd5fa902500 bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb9e0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb97cd1657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb9e047fb902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb99c083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb97cc4fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb9e88a637ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb95cd5fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3199) Received paxos partition sync request from bb9a4083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3228) All partition data has been received by principal Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb9e88a637ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb9e0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb9e047fb902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb9a4083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb99c083a7ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb97cd1657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb97cc4fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb95cd5fa902500 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3231) Sending paxos partition sync message to bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2447) CLUSTER SIZE = 10 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2485) Global state is well formed Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2230) setting replication factors: cluster size 10, paxos single replica limit 1 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2245) {test} replication factor is 2 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2245) {bidder} replication factor is 2 Jun 22 2017 23:14:20 GMT: INFO (config): (cluster_config.c:421) rack aware is disabled Jun 22 2017 23:14:20 GMT: INFO (partition): (cluster_config.c:380) rack aware is disabled Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:3222) {test} re-balanced, expected migrations - (86 tx, 86 rx) Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:3222) {bidder} re-balanced, expected migrations - (86 tx, 86 rx) Jun 22 2017 23:14:21 GMT: INFO (paxos): (partition.c:3239) global partition state: total 8192 lost 0 unique 6604 duplicate 1588 Jun 22 2017 23:14:21 GMT: INFO (paxos): (partition.c:3241) partition state after fixing lost partitions (master): total 8192 lost 0 unique 6604 duplicate 1588 Jun 22 2017 23:14:21 GMT: INFO (paxos): (partition.c:3243) 0 new partition version tree paths generated Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:348) ALLOW MIGRATIONS ``` **Node BB9A4083A7AC40C (this node seems restarted) :** ``` Jun 22 2017 23:14:18 GMT: INFO (hb): (hb.c:2352) hb considers expiring: now 7918874972 last 7918873450 Jun 22 2017 23:14:18 GMT: INFO (hb): (hb.c:2371) hb expires but fabric says DEAD: node bb9f0d0657ac40c Jun 22 2017 23:14:18 GMT: INFO (hb): (hb.c:2550) removing node on heartbeat failure: bb9f0d0657ac40c Jun 22 2017 23:14:18 GMT: INFO (fabric): (fabric.c:1762) fabric: node bb9f0d0657ac40c departed Jun 22 2017 23:14:18 GMT: INFO (fabric): (fabric.c:1678) fabric disconnecting node: bb9f0d0657ac40c Jun 22 2017 23:14:18 GMT: INFO (fabric): (fabric.c:1288) can't write to notification file descriptor: will probably have to take down process Jun 22 2017 23:14:18 GMT: INFO (fabric): (fabric.c:1288) can't write to notification file descriptor: will probably have to take down process Jun 22 2017 23:14:18 GMT: INFO (paxos): (paxos.c:2175) departed node bb9f0d0657ac40c is no longer principal pro tempore Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4812) system memory: free 17107860kb ( 25 percent free ) Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4818) ClusterSize 10 ::: objects 221429046 ::: sub_objects 0 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4827) rec refs 221598435 ::: rec locks 0 ::: trees 0 ::: wr reqs 0 ::: mig tx 0 ::: mig rx 41 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4832) replica errs :: null 0 non-null 0 ::: sync copy errs :: master 0 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7118, 582128, 575010) : hb (14, 2139, 2125) : fab (133, 508, 375) Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4844) heartbeat_received: self 0 : foreign 22867939 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4845) heartbeat_stats: bt 0 bf 20397534 nt 0 ni 0 nn 0 nnir 0 nal 0 sf1 0 sf2 0 sf3 0 sf4 0 sf5 0 sf6 1 mrf 0 eh 180 efd 30 efa 150 um 8536 mcf 0 rc 2125 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4857) tree_counts: nsup 1 scan 0 dup 0 wprocess 0 migrx 41 migtx 0 ssdr 0 ssdw 0 rw 0 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4874) {test} disk bytes used 0 : avail pct 100 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4876) {test} memory bytes used 0 (index 0 : sindex 0 : data 0) : used pct 0.00 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4924) {test} migrations - complete Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4874) {bidder} disk bytes used 60041106048 : avail pct 92 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4876) {bidder} memory bytes used 38546442099 (index 14171458944 : sindex 97843 : data 24374885312) : used pct 71.80 Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4924) {bidder} migrations - complete Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4931) partitions: actual 780 sync 808 desync 0 zombie 0 absent 6604 Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:137) histogram dump: writes_master (829642 total) msec Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (00: 0000806699) (01: 0000007724) (02: 0000002372) (03: 0000002318) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (04: 0000003673) (05: 0000001484) (06: 0000000287) (07: 0000000183) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (08: 0000000148) (09: 0000000141) (10: 0000000545) (11: 0000003647) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:163) (12: 0000000345) (13: 0000000032) (14: 0000000044) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:137) histogram dump: proxy (723 total) msec Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (00: 0000000170) (01: 0000000085) (02: 0000000070) (03: 0000000076) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (04: 0000000052) (05: 0000000059) (06: 0000000047) (07: 0000000044) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:163) (08: 0000000057) (09: 0000000035) (10: 0000000028) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:137) histogram dump: udf (79910506 total) msec Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (00: 0079720787) (01: 0000121231) (02: 0000042560) (03: 0000017079) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (04: 0000004808) (05: 0000002060) (06: 0000001409) (07: 0000000354) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (08: 0000000028) (09: 0000000023) (10: 0000000058) (11: 0000000063) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:163) (12: 0000000041) (13: 0000000004) (14: 0000000001) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:137) histogram dump: query (795945 total) msec Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (00: 0000012866) (01: 0000384558) (02: 0000376542) (03: 0000015417) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (04: 0000003215) (05: 0000000816) (06: 0000001260) (07: 0000000598) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (08: 0000000195) (09: 0000000099) (10: 0000000123) (11: 0000000138) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:163) (12: 0000000066) (13: 0000000038) (14: 0000000014) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:137) histogram dump: query_rec_count (783195 total) count Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (01: 0000001603) (02: 0000000092) (03: 0000000535) (04: 0000000897) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:154) (05: 0000000232) (06: 0000002333) (07: 0000008402) (08: 0000768791) Jun 22 2017 23:14:19 GMT: INFO (info): (hist.c:163) (09: 0000000310) Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2352) hb considers expiring: now 7918876927 last 7918875414 Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2380) hb expires but fabric says ALIVE: lasttime 161 node bb95cd5fa902500 Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:1683) as_hb_tcp_recv() fd 5012 returning error try 3 len 339 read_so_far 184 Jun 22 2017 23:14:20 GMT: WARNING (hb): (hb.c:2227) about to goto CloseSocket.... Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:1070) initiated connection to mesh seed host at 10.56.101.158:3002 (10.56.101.158:3002) via socket 5671 from 10.57.115.42:57674 Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2352) hb considers expiring: now 7918877077 last 7918875414 Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2380) hb expires but fabric says ALIVE: lasttime 321 node bb95cd5fa902500 Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2352) hb considers expiring: now 7918877228 last 7918875414 Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:405) Sync message received from a principal bb9f0d0657ac40c that is back from the dead? Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2380) hb expires but fabric says ALIVE: lasttime 442 node bb95cd5fa902500 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:356) DISALLOW MIGRATIONS Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2352) hb considers expiring: now 7918877228 last 7918875628 Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2380) hb expires but fabric says ALIVE: lasttime 442 node bb9e88a637ac40c Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2556) new heartbeat received: bb9f0d0657ac40c principal node is bb9f0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (fabric): (fabric.c:1756) fabric: node bb9f0d0657ac40c arrived Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3168) SUCCESSION [34]@bb9f0d0657ac40c*: bb9f0d0657ac40c bb9e88a637ac40c bb9e0d0657ac40c bb9e047fb902500 bb9a4083a7ac40c bb99c083a7ac40c bb97cd1657ac40c bb97cc4fa902500 bb95cd5fa902500 bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3179) node bb9f0d0657ac40c is now principal pro tempore Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:2305) Sent partition sync request to node bb9f0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:1683) as_hb_tcp_recv() fd 7458 returning error try 3 len 339 read_so_far 184 Jun 22 2017 23:14:20 GMT: WARNING (hb): (hb.c:2227) about to goto CloseSocket.... Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:1683) as_hb_tcp_recv() fd 993 returning error try 3 len 339 read_so_far 184 Jun 22 2017 23:14:20 GMT: WARNING (hb): (hb.c:2227) about to goto CloseSocket.... Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:1683) as_hb_tcp_recv() fd 186 returning error try 3 len 339 read_so_far 184 Jun 22 2017 23:14:20 GMT: WARNING (hb): (hb.c:2227) about to goto CloseSocket.... Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2352) hb considers expiring: now 7918877380 last 7918875628 Jun 22 2017 23:14:20 GMT: INFO (hb): (hb.c:2380) hb expires but fabric says ALIVE: lasttime 602 node bb9e88a637ac40c Jun 22 2017 23:14:21 GMT: INFO (hb): (hb.c:1070) initiated connection to mesh non-seed host at 10.100.15.18:3002 (10.100.15.18:3002) via socket 186 from 10.57.115.42:58952 Jun 22 2017 23:14:21 GMT: INFO (fabric): (fabric.c:1318) can't write to notification file descriptor: will probably have to take down process Jun 22 2017 23:14:21 GMT: INFO (hb): (hb.c:1683) as_hb_tcp_recv() fd 7598 returning error try 3 len 339 read_so_far 184 Jun 22 2017 23:14:21 GMT: WARNING (hb): (hb.c:2227) about to goto CloseSocket.... Jun 22 2017 23:14:21 GMT: INFO (hb): (hb.c:1070) initiated connection to mesh non-seed host at 10.100.15.96:3002 (10.100.15.96:3002) via socket 5740 from 10.57.115.42:42176 Jun 22 2017 23:14:21 GMT: INFO (paxos): (paxos.c:3263) received partition sync message from bb9f0d0657ac40c Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2447) CLUSTER SIZE = 10 Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2485) Global state is well formed Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2230) setting replication factors: cluster size 10, paxos single replica limit 1 Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2245) {test} replication factor is 2 Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2245) {bidder} replication factor is 2 Jun 22 2017 23:14:21 GMT: INFO (config): (cluster_config.c:421) rack aware is disabled Jun 22 2017 23:14:21 GMT: INFO (partition): (cluster_config.c:380) rack aware is disabled Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:3222) {test} re-balanced, expected migrations - (794 tx, 794 rx) Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:3222) {bidder} re-balanced, expected migrations - (794 tx, 794 rx) Jun 22 2017 23:14:21 GMT: INFO (paxos): (partition.c:3239) global partition state: total 8192 lost 0 unique 6604 duplicate 1588 Jun 22 2017 23:14:21 GMT: INFO (paxos): (partition.c:3241) partition state after fixing lost partitions (master): total 8192 lost 0 unique 6604 duplicate 1588 Jun 22 2017 23:14:21 GMT: INFO (paxos): (partition.c:3243) 0 new partition version tree paths generated Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:348) ALLOW MIGRATIONS Jun 22 2017 23:14:23 GMT: INFO (fabric): (fabric.c:1318) can't write to notification file descriptor: will probably have to take down process Jun 22 2017 23:14:23 GMT: INFO (hb): (hb.c:1070) initiated connection to mesh seed host at 10.56.101.152:3002 (10.56.101.152:3002) via socket 5756 from 10.57.115.42:49250 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4812) system memory: free 17138988kb ( 26 percent free ) Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4818) ClusterSize 10 ::: objects 221429068 ::: sub_objects 0 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4827) rec refs 221490160 ::: rec locks 0 ::: trees 0 ::: wr reqs 8 ::: mig tx 1137 ::: mig rx 460 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4832) replica errs :: null 0 non-null 0 ::: sync copy errs :: master 0 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 8 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7269, 582289, 575020) : hb (18, 2148, 2130) : fab (142, 524, 382) Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4844) heartbeat_received: self 0 : foreign 22868988 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4845) heartbeat_stats: bt 0 bf 20398259 nt 0 ni 0 nn 0 nnir 0 nal 0 sf1 0 sf2 0 sf3 0 sf4 0 sf5 0 sf6 1 mrf 0 eh 185 efd 30 efa 155 um 8536 mcf 0 rc 2130 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4857) tree_counts: nsup 1 scan 0 dup 0 wprocess 0 migrx 460 migtx 1137 ssdr 0 ssdw 0 rw 8 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4874) {test} disk bytes used 0 : avail pct 100 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4876) {test} memory bytes used 0 (index 0 : sindex 0 : data 0) : used pct 0.00 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4922) {test} migrations - remaining (733 tx, 384 rx), active (1 tx, 10 rx), 29.66% complete Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4874) {bidder} disk bytes used 60041114752 : avail pct 92 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4876) {bidder} memory bytes used 38546446467 (index 14171460352 : sindex 97843 : data 24374888272) : used pct 71.80 Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4922) {bidder} migrations - remaining (794 tx, 794 rx), active (1 tx, 10 rx), 0.00% complete ``` continue … ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 23, 2017, 9:00am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/8 "2017-06-23T09:00:33Z")
Node BB9E047FB902500 : ``` Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:2919) {32} sending prepare_ack to bb9f0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:356) DISALLOW MIGRATIONS Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3168) SUCCESSION [33]@bb9f0d0657ac40c*: bb9f0d0657ac40c bb9e88a637ac40c bb9e0d0657ac40c bb9e047fb902500 bb99c083a7ac40c bb97cd1657ac40c bb97cc4fa902500 bb95cd5fa902500 bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3179) node bb9f0d0657ac40c is now principal pro tempore Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:2305) Sent partition sync request to node bb9f0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3263) received partition sync message from bb9f0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2447) CLUSTER SIZE = 9 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2485) Global state is well formed Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2230) setting replication factors: cluster size 9, paxos single replica limit 1 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2245) {test} replication factor is 2 Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:2245) {bidder} replication factor is 2 Jun 22 2017 23:14:20 GMT: INFO (config): (cluster_config.c:421) rack aware is disabled Jun 22 2017 23:14:20 GMT: INFO (partition): (cluster_config.c:380) rack aware is disabled Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:3222) {test} re-balanced, expected migrations - (72 tx, 92 rx) Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:3222) {bidder} re-balanced, expected migrations - (72 tx, 92 rx) Jun 22 2017 23:14:20 GMT: INFO (paxos): (partition.c:3239) global partition state: total 8192 lost 0 unique 8192 duplicate 0 Jun 22 2017 23:14:20 GMT: INFO (paxos): (partition.c:3241) partition state after fixing lost partitions (master): total 8192 lost 0 unique 8192 duplicate 0 Jun 22 2017 23:14:20 GMT: INFO (paxos): (partition.c:3243) 1588 new partition version tree paths generated Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:348) ALLOW MIGRATIONS Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:2919) {33} sending prepare_ack to bb9f0d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (partition): (partition.c:356) DISALLOW MIGRATIONS Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3168) SUCCESSION [34]@bb9f0d0657ac40c*: bb9f0d0657ac40c bb9e88a637ac40c bb9e0d0657ac40c bb9e047fb902500 bb9a4083a7ac40c bb99c083a7ac40c bb97cd1657ac40c bb97cc4fa902500 bb95cd5fa902500 bb904d0657ac40c Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:3179) node bb9f0d0657ac40c is now principal pro tempore Jun 22 2017 23:14:20 GMT: INFO (paxos): (paxos.c:2305) Sent partition sync request to node bb9f0d0657ac40c Jun 22 2017 23:14:21 GMT: INFO (paxos): (paxos.c:3263) received partition sync message from bb9f0d0657ac40c Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2447) CLUSTER SIZE = 10 Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2485) Global state is well formed Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2230) setting replication factors: cluster size 10, paxos single replica limit 1 Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2245) {test} replication factor is 2 Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:2245) {bidder} replication factor is 2 Jun 22 2017 23:14:21 GMT: INFO (config): (cluster_config.c:421) rack aware is disabled Jun 22 2017 23:14:21 GMT: INFO (partition): (cluster_config.c:380) rack aware is disabled Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:3222) {test} re-balanced, expected migrations - (72 tx, 72 rx) Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:3222) {bidder} re-balanced, expected migrations - (72 tx, 72 rx) Jun 22 2017 23:14:21 GMT: INFO (paxos): (partition.c:3239) global partition state: total 8192 lost 0 unique 6604 duplicate 1588 Jun 22 2017 23:14:21 GMT: INFO (paxos): (partition.c:3241) partition state after fixing lost partitions (master): total 8192 lost 0 unique 6604 duplicate 1588 Jun 22 2017 23:14:21 GMT: INFO (paxos): (partition.c:3243) 0 new partition version tree paths generated Jun 22 2017 23:14:21 GMT: INFO (partition): (partition.c:348) ALLOW MIGRATIONS Jun 22 2017 23:14:21 GMT: WARNING (fabric): (fabric.c:2323) No fabric transmit structure in global hash for fabric transaction-id 38 Jun 22 2017 23:14:21 GMT: INFO (info): (thr_info.c:4812) system memory: free 15330404kb ( 23 percent free ) Jun 22 2017 23:14:21 GMT: INFO (info): (thr_info.c:4818) ClusterSize 10 ::: objects 224981662 ::: sub_objects 0 ``` what could be the reason? Is it possible to return incorrect data during this period ? I have seen some inconsistency (key present but returned not present) during this period. Can it be because of this? ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [June 23, 2017, 4:21pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/9 "2017-06-23T16:21:19Z")
Looks like the clustering algorithms were in a bad state. [3.7.0.1](http://www.aerospike.com/download/server/notes.html#3.7.0.1) introduced a new paxos recovery policy which later became default, but not sure when. Make sure your cluster is using this policy as it eliminates a lot of these problems. Yes, in this case the cluster will behave as if there were a network partition, inconsistent operations are expected. 3.13 introduced a new clustering system which addresses additional issues that were not possible to address with a recovery policy. ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 23, 2017, 4:39pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/10 "2017-06-23T16:39:34Z")
I am using 3.8.1 version. How can i apply that policy? do you have any document on this? ---
**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\ **Post date:** [June 24, 2017, 7:07am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/11 "2017-06-24T07:07:40Z")
I think @kporter is referring to paxos-recovery-policy=auto-reset-master. It’s helped stabilize my clusters greatly 😃 ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 24, 2017, 9:29am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/12 "2017-06-24T09:29:51Z")
According to this document [Configuration Reference | Aerospike Documentation](http://www.aerospike.com/docs/reference/configuration#paxos-recovery-policy) It seems that auto-reset-master is already default as i am using 3.8.1 version. ---
**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\ **Post date:** [June 25, 2017, 8:54pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/13 "2017-06-25T20:54:34Z")
You can validate this is your policy by running asinfo -v ‘get-config:context=service’ -l | grep -i paxos-rec ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 26, 2017, 7:09am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/14 "2017-06-26T07:09:26Z")
Yes, it is auto-reset-master. ``` $ asinfo -v 'get-config:context=service' -l | grep -i paxos-rec paxos-recovery-policy=auto-reset-master ``` ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [June 26, 2017, 5:53pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/15 "2017-06-26T17:53:52Z")
The old clustering algorithm still had flaws even with this policy; however, they were very difficult to exercise (the remaining issues were addressed in paxos-protocol v5 in 3.13.0 release. With this policy running I would begin to more strongly suspect a network problem. ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 26, 2017, 6:19pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/16 "2017-06-26T18:19:20Z")
@kporter i have posted one more question couple of days ago but didn’t get any response yet. > [@Aerospike Go client: timed out issue](https://discuss.aerospike.com/t/aerospike-go-client-timed-out-issue/4292): > > I am using Aerospike go-client (v1.24.0). I have 10 node aerospike cluster (v3.8.1). I am getting frequent error message - E0621 09:59:01.457130 2514 Aerospike.go:32] Unable to fetch from Aerospike : command execution timed out: Exceeded number of retries. See Policy.MaxRetries. (last error: No available connections to the node. Connection Pool was empty, and limited to certain number of connections.) There are some read connection timeout logs also: E0621 10:22:06.522898 22038 Aerospike.go:… Do you think that it is also related to this? If not, can you guide me in right direction? ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [June 26, 2017, 6:34pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/17 "2017-06-26T18:34:47Z")
Seems to also correlate with network problems. Could you share the log lines containing the work “proto” about 5 min prior and a few minutes into one of these events? ```auto grep "proto" /var/log/aerospike/aerospike.log ``` ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 27, 2017, 5:02am UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/18 "2017-06-27T05:02:18Z")
@kporter this timeout issue is very frequent (multiple times a day). this was present even before migration issue and still present today. > Seems to also correlate with network problems Are you referring to Aerospike clustering algorithm or somethiing related to my cluster network? ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [June 27, 2017, 5:51pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/19 "2017-06-27T17:51:10Z")
> [@nishant.nitjsr](#): > > Are you referring to Aerospike clustering algorithm or something related to my cluster network? Yes, all the above. These log warnings aren’t part of normal operations based on information provided so far I would lean more towards the actual network itself. BTW, would be interesting to see the “proto” log lines I mentioned. ---
**Author:** ![nishant.nitjsr](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/nishant.nitjsr/32/501_2.png) [@nishant.nitjsr](https://discuss.aerospike.com/u/nishant.nitjsr)\ **Post date:** [June 28, 2017, 12:09pm UTC](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289/20 "2017-06-28T12:09:55Z")
`grep "proto" /var/log/aerospike/aerospike.log-20170623` on Node BB9A4083A7AC40C: **Output:** ``` Jun 22 2017 23:12:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (6985, 581275, 574290) : hb (16, 2135, 2119) : fab (140, 508, 368) Jun 22 2017 23:12:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (6984, 581277, 574293) : hb (16, 2135, 2119) : fab (140, 508, 368) Jun 22 2017 23:13:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7057, 581665, 574608) : hb (15, 2138, 2123) : fab (140, 508, 368) Jun 22 2017 23:13:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 2 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7070, 581686, 574616) : hb (16, 2139, 2123) : fab (140, 508, 368) Jun 22 2017 23:13:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7069, 581690, 574621) : hb (16, 2139, 2123) : fab (140, 508, 368) Jun 22 2017 23:13:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7064, 581697, 574633) : hb (16, 2139, 2123) : fab (140, 508, 368) Jun 22 2017 23:13:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 9 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7293, 581974, 574681) : hb (16, 2139, 2123) : fab (140, 508, 368) Jun 22 2017 23:13:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7311, 581994, 574683) : hb (16, 2139, 2123) : fab (140, 508, 368) Jun 22 2017 23:14:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (6998, 581998, 575000) : hb (16, 2139, 2123) : fab (140, 508, 368) Jun 22 2017 23:14:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7118, 582128, 575010) : hb (14, 2139, 2125) : fab (133, 508, 375) Jun 22 2017 23:14:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 8 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7269, 582289, 575020) : hb (18, 2148, 2130) : fab (142, 524, 382) Jun 22 2017 23:14:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 16 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7269, 582316, 575047) : hb (18, 2148, 2130) : fab (142, 524, 382) Jun 22 2017 23:14:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 201 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7570, 582736, 575166) : hb (18, 2148, 2130) : fab (142, 524, 382) Jun 22 2017 23:14:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 149 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7530, 583334, 575804) : hb (18, 2148, 2130) : fab (142, 524, 382) Jun 22 2017 23:15:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 181 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7538, 583865, 576327) : hb (18, 2148, 2130) : fab (142, 524, 382) Jun 22 2017 23:15:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 186 prox 0 wait 1 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7540, 584527, 576987) : hb (18, 2149, 2131) : fab (142, 524, 382) Jun 22 2017 23:15:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 143 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7531, 585094, 577563) : hb (18, 2149, 2131) : fab (142, 524, 382) Jun 22 2017 23:15:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 169 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7471, 585568, 578097) : hb (18, 2150, 2132) : fab (142, 524, 382) Jun 22 2017 23:15:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 213 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7544, 586334, 578790) : hb (18, 2150, 2132) : fab (142, 524, 382) Jun 22 2017 23:15:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 135 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7517, 587007, 579490) : hb (18, 2150, 2132) : fab (142, 524, 382) Jun 22 2017 23:16:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 234 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7522, 587635, 580113) : hb (18, 2150, 2132) : fab (142, 524, 382) Jun 22 2017 23:16:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 155 prox 0 wait 4 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7410, 588373, 580963) : hb (18, 2150, 2132) : fab (142, 524, 382) Jun 22 2017 23:16:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 94 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7337, 588887, 581550) : hb (18, 2150, 2132) : fab (142, 524, 382) Jun 22 2017 23:16:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 29 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7275, 589102, 581827) : hb (18, 2150, 2132) : fab (142, 524, 382) Jun 22 2017 23:16:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 7 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7362, 589307, 581945) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:16:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 12 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7363, 589350, 581987) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:17:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 25 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7362, 589395, 582033) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:17:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 4 prox 0 wait 2 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7335, 589410, 582075) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:17:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 5 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7334, 589450, 582116) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:17:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 11 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7344, 589493, 582149) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:17:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7325, 589527, 582202) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:17:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 13 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7348, 589568, 582220) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:18:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 5 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7325, 589607, 582282) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:18:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 11 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7338, 589650, 582312) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:18:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 4 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7320, 589681, 582361) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:18:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7319, 589689, 582370) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:18:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 8 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7334, 589728, 582394) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:18:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 18 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7344, 589753, 582409) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:19:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7323, 589763, 582440) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:19:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 16 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7337, 589798, 582461) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:19:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7321, 589817, 582496) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:19:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 14 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7337, 589849, 582512) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:19:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7323, 589864, 582541) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:19:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 13 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7341, 589899, 582558) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:20:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 6 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7372, 590005, 582633) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:20:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7517, 590182, 582665) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:20:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 15 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7532, 590230, 582698) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:20:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7520, 590244, 582724) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:20:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 11 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7529, 590290, 582761) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:20:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7519, 590296, 582777) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:21:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7518, 590321, 582803) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:21:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 13 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7512, 590345, 582833) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:21:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 17 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7522, 590379, 582857) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:21:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 19 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7526, 590409, 582883) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:21:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 4 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7516, 590418, 582902) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:21:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 11 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7523, 590458, 582935) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:22:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7508, 590487, 582979) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:22:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7507, 590501, 582994) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:22:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 7 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7529, 590570, 583041) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:22:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 16 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7536, 590590, 583054) : hb (17, 2150, 2133) : fab (142, 524, 382) Jun 22 2017 23:22:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7540, 590612, 583072) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:22:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7527, 590623, 583096) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:23:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7546, 590669, 583123) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:23:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 8 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7560, 590699, 583139) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:23:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 20 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7549, 590717, 583168) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:23:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7538, 590720, 583182) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:23:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7554, 590754, 583200) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:23:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 17 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7584, 590792, 583208) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:24:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7572, 590805, 583233) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:24:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 3 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7572, 590838, 583266) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:24:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 11 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7611, 590885, 583274) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:24:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 5 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7619, 590931, 583312) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:24:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 4 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7640, 590982, 583342) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:24:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 18 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7654, 591001, 583347) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:25:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7650, 591008, 583358) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:25:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 6 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7691, 591080, 583389) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:25:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7691, 591085, 583394) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:25:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7693, 591093, 583400) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:25:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7660, 591103, 583443) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:25:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7667, 591111, 583444) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:26:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7640, 591120, 583480) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:26:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7657, 591142, 583485) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:26:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7654, 591149, 583495) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:26:39 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7661, 591158, 583497) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:26:49 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7678, 591184, 583506) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:26:59 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7680, 591198, 583518) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:27:09 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7678, 591210, 583532) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:27:19 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7679, 591221, 583542) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:27:29 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7679, 591222, 583543) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:27:40 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7678, 591224, 583546) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:27:50 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7678, 591232, 583554) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:28:00 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7678, 591232, 583554) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:28:10 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7678, 591238, 583560) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:28:20 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7676, 591243, 583567) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:28:30 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7676, 591247, 583571) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:28:40 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7677, 591249, 583572) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:28:50 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7676, 591255, 583579) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:29:00 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7677, 591258, 583581) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:29:10 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7673, 591266, 583593) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:29:20 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7673, 591274, 583601) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:29:30 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7673, 591275, 583602) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:29:40 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7674, 591281, 583607) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:29:50 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7677, 591311, 583634) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:30:00 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7682, 591317, 583635) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:30:10 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 6 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7715, 591354, 583639) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:30:20 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 6 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7715, 591360, 583645) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:30:30 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7716, 591369, 583653) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:30:40 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 3 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7714, 591370, 583656) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:30:50 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 3 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7715, 591380, 583665) : hb (17, 2152, 2135) : fab (142, 524, 382) Jun 22 2017 23:31:00 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (7715, 591381, 583666) : hb (17, 2152, 2135) : fab (142, 524, 382) ``` On BB9E047FB902500 node: ``` Jun 22 2017 23:13:21 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4440, 10622224, 10617784) : hb (16, 1979, 1963) : fab (141, 225, 84) Jun 22 2017 23:13:31 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4441, 10622231, 10617790) : hb (16, 1979, 1963) : fab (141, 225, 84) Jun 22 2017 23:13:41 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4434, 10622234, 10617800) : hb (16, 1979, 1963) : fab (141, 225, 84) Jun 22 2017 23:13:51 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4564, 10622402, 10617838) : hb (16, 1979, 1963) : fab (141, 225, 84) Jun 22 2017 23:14:01 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4564, 10622407, 10617843) : hb (16, 1979, 1963) : fab (141, 225, 84) Jun 22 2017 23:14:11 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4399, 10622421, 10618022) : hb (16, 1979, 1963) : fab (141, 225, 84) Jun 22 2017 23:14:21 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4400, 10622424, 10618024) : hb (15, 1979, 1964) : fab (141, 225, 84) Jun 22 2017 23:14:31 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4400, 10622430, 10618030) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:14:41 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4400, 10622434, 10618034) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:14:51 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 5 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4498, 10622548, 10618050) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:15:01 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4507, 10622562, 10618055) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:15:11 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 17 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4523, 10622581, 10618058) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:15:21 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 6 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4534, 10622604, 10618070) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:15:31 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4534, 10622609, 10618075) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:15:41 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4524, 10622619, 10618095) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:15:51 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 15 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4534, 10622645, 10618111) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:16:01 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 7 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4509, 10622653, 10618144) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:16:11 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 10 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4530, 10622674, 10618144) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:16:21 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 11 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4517, 10622677, 10618160) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:16:31 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 2 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4508, 10622692, 10618184) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:16:41 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4508, 10622695, 10618187) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:16:51 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4465, 10622699, 10618234) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:17:01 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4431, 10622715, 10618284) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:17:11 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4411, 10622731, 10618320) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:17:21 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 1 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4400, 10622738, 10618338) : hb (16, 1980, 1964) : fab (142, 226, 84) Jun 22 2017 23:17:31 GMT: INFO (info): (thr_info.c:4842) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: iq 0 ::: dq 0 : fds - proto (4401, 10622743, 10618342) : hb (16, 1980, 1964) : fab (142, 226, 84) ``` [Next page](https://discuss.aerospike.com/t/urgent-migration-stuck-v3-8-1-missing-acks-from-node/4289.md?page=2)