Increase Timeout for asrestore

qorbani · March 17, 2016, 11:53pm

I’m trying to restore 0.5TB data using “asrestore”. It was restoring fine until it hit around 55-60% completion but then I started to receiving lots of errors:

2016-03-17 23:45:07 GMT [ERR] [25401] Error while storing record - code 9: Client timeout: timeout=10000 iterations=1 failedNodes=0 failedConns=0 at src/main/aerospike/as_command.c:543
2016-03-17 23:45:08 GMT [VER] [25390] 24014 per-thread record(s) (0 rec/s), read latency: 7 (3) us, store latency: 160017 (1481595) us
2016-03-17 23:45:08 GMT [VER] [25402] 37044 per-thread record(s) (1 rec/s), read latency: 17 (4) us, store latency: 149220 (905100) us
2016-03-17 23:45:09 GMT [VER] [25386] 58933 per-thread record(s) (2 rec/s), read latency: 14 (39) us, store latency: 83162 (421030) us
2016-03-17 23:45:09 GMT [VER] [25388] 76910 per-thread record(s) (2 rec/s), read latency: 12 (3) us, store latency: 69489 (317434) us
2016-03-17 23:45:09 GMT [VER] [25395] 18997 per-thread record(s) (0 rec/s), read latency: 6 (4) us, store latency: 194527 (1313947) us
2016-03-17 23:45:10 GMT [VER] [25394] 273316 per-thread record(s) (0 rec/s), read latency: 14 (3) us, store latency: 21168 (2776697) us
2016-03-17 23:45:10 GMT [INF] [25226] 0 UDF file(s), 0 secondary index(es), 749229341 record(s) (17 KiB/s, 40 rec/s, 442 B/rec, backed off: 0)
2016-03-17 23:45:10 GMT [INF] [25226] Expired 0 : skipped 0 : inserted 749228854 : failed 487 (existed 0, fresher 487)
2016-03-17 23:45:10 GMT [INF] [25226] 60% complete, ~>100d remaining
2016-03-17 23:45:10 GMT [VER] [25405] 57612 per-thread record(s) (2 rec/s), read latency: 11 (3) us, store latency: 67097 (423176) us
2016-03-17 23:45:10 GMT [VER] [25404] 65503 per-thread record(s) (2 rec/s), read latency: 13 (3) us, store latency: 66131 (432556) us
2016-03-17 23:45:11 GMT [VER] [25399] 66676 per-thread record(s) (2 rec/s), read latency: 12 (3) us, store latency: 69980 (401560) us
2016-03-17 23:45:12 GMT [VER] [25397] 134079 per-thread record(s) (6 rec/s), read latency: 12 (3) us, store latency: 29101 (104509) us
2016-03-17 23:45:12 GMT [ERR] [25403] Error while storing record - code 9: Client timeout: timeout=10000 iterations=1 failedNodes=0 failedConns=0 at src/main/aerospike/as_command.c:543

Restore stuck in 60% for hours now and not moving ahead. In AMC, I only see 20-40 writes per seconds now while it was 50,000 writes. Every 1-2 minutes write spikes to 1-2k and then drop back to 20-40 again! It looks like client timing out but I don’t see any option to define timeout for asrestore.

Is there way to increase default timeout?

anushree · March 18, 2016, 12:43am

There is no option to increase the default timeout for asrestore.

Are you seeing latencies on your cluster? Is there sufficient capacity? Any other network issues between where you are restoring from an your server? Or are you restoring locally?

Can you paste the output of the following:

asadm > info
asadm > show latency

qorbani · March 18, 2016, 1:34am

info node:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Service Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                      Node     Build   Cluster            Cluster      Cluster     Cluster    Free   Free    Migrates                                    Principal     Objects     Uptime
                                         .         .      Size                Key   Visibility   Integrity   Disk%   Mem%   (tx,rx,a)                                            .           .          .
usc1b-edge-aero-01.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   220.884 M   26:55:47
usc1b-edge-aero-02.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.082 M   26:41:55
usc1b-edge-aero-03.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.097 M   26:40:23
usc1b-edge-aero-04.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.119 M   26:41:23
usc1b-edge-aero-05.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.401 M   26:41:16
usc1b-edge-aero-06.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.028 M   26:38:51
usc1b-edge-aero-07.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.283 M   26:40:55
usc1b-edge-aero-08.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.366 M   26:39:37
usc1b-edge-aero-09.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.388 M   26:40:21
usc1b-edge-aero-10.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.426 M   26:40:09
usc1b-edge-aero-11.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.429 M   26:40:08
usc1b-edge-aero-12.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.055 M   26:40:23
usc1b-edge-aero-13.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   220.911 M   26:39:00
usc1b-edge-aero-14.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.291 M   26:40:03
usc1b-edge-aero-15.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   220.998 M   26:36:37
usc1b-edge-aero-16.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.285 M   26:39:40
usc1b-edge-aero-17.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.232 M   26:39:55
usc1b-edge-aero-18.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.265 M   26:34:36
usc1b-edge-aero-19.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   220.935 M   26:39:21
usc1b-edge-aero-20.c.bc-edge.internal:3000   3.7.4.1        20   FA4C837287FAAA4E   True         True           82     40   (0,0,0)     usc1b-edge-aero-20.c.bc-edge.internal:3000   221.088 M   26:39:35
Number of rows: 20

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                      Node               Node                                         Fqdn                 Ip   Client     Current     HB         HB
                                         .                 Id                                            .                  .    Conns        Time   Self    Foreign
usc1b-edge-aero-01.c.bc-edge.internal:3000   BB902140A0A0142    usc1b-edge-aero-01.c.bc-edge.internal:3000   10.10.20.2:3000        21   195960762      0   23994339
usc1b-edge-aero-02.c.bc-edge.internal:3000   BB903140A0A0142    usc1b-edge-aero-02.c.bc-edge.internal:3000   10.10.20.3:3000        17   195960762      0   23994310
usc1b-edge-aero-03.c.bc-edge.internal:3000   BB904140A0A0142    usc1b-edge-aero-03.c.bc-edge.internal:3000   10.10.20.4:3000        16   195960762      0   23360098
usc1b-edge-aero-04.c.bc-edge.internal:3000   BB905140A0A0142    usc1b-edge-aero-04.c.bc-edge.internal:3000   10.10.20.5:3000        16   195960762      0   23993850
usc1b-edge-aero-05.c.bc-edge.internal:3000   BB906140A0A0142    usc1b-edge-aero-05.c.bc-edge.internal:3000   10.10.20.6:3000        16   195960762      0   24625587
usc1b-edge-aero-06.c.bc-edge.internal:3000   BB907140A0A0142    usc1b-edge-aero-06.c.bc-edge.internal:3000   10.10.20.7:3000        17   195960762      0   20192539
usc1b-edge-aero-07.c.bc-edge.internal:3000   BB908140A0A0142    usc1b-edge-aero-07.c.bc-edge.internal:3000   10.10.20.8:3000        16   195960762      0   23993077
usc1b-edge-aero-08.c.bc-edge.internal:3000   BB909140A0A0142    usc1b-edge-aero-08.c.bc-edge.internal:3000   10.10.20.9:3000        17   195960762      0   25250679
usc1b-edge-aero-09.c.bc-edge.internal:3000   BB90A140A0A0142    usc1b-edge-aero-09.c.bc-edge.internal:3000   10.10.20.10:3000       17   195960762      0   25253601
usc1b-edge-aero-10.c.bc-edge.internal:3000   BB90B140A0A0142    usc1b-edge-aero-10.c.bc-edge.internal:3000   10.10.20.11:3000       17   195960762      0   25253485
usc1b-edge-aero-11.c.bc-edge.internal:3000   BB90C140A0A0142    usc1b-edge-aero-11.c.bc-edge.internal:3000   10.10.20.12:3000       16   195960762      0   24621851
usc1b-edge-aero-12.c.bc-edge.internal:3000   BB90D140A0A0142    usc1b-edge-aero-12.c.bc-edge.internal:3000   10.10.20.13:3000       30   195960762      0   25254473
usc1b-edge-aero-13.c.bc-edge.internal:3000   BB90E140A0A0142    usc1b-edge-aero-13.c.bc-edge.internal:3000   10.10.20.14:3000       19   195960762      0   22085874
usc1b-edge-aero-14.c.bc-edge.internal:3000   BB90F140A0A0142    usc1b-edge-aero-14.c.bc-edge.internal:3000   10.10.20.15:3000       16   195960762      0   23989761
usc1b-edge-aero-15.c.bc-edge.internal:3000   BB910140A0A0142    usc1b-edge-aero-15.c.bc-edge.internal:3000   10.10.20.16:3000       17   195960762      0   22687180
usc1b-edge-aero-16.c.bc-edge.internal:3000   BB911140A0A0142    usc1b-edge-aero-16.c.bc-edge.internal:3000   10.10.20.17:3000       17   195960762      0   23986739
usc1b-edge-aero-17.c.bc-edge.internal:3000   BB912140A0A0142    usc1b-edge-aero-17.c.bc-edge.internal:3000   10.10.20.18:3000       17   195960762      0   27144868
usc1b-edge-aero-18.c.bc-edge.internal:3000   BB913140A0A0142    usc1b-edge-aero-18.c.bc-edge.internal:3000   10.10.20.19:3000       19   195960762      0   22029812
usc1b-edge-aero-19.c.bc-edge.internal:3000   BB914140A0A0142    usc1b-edge-aero-19.c.bc-edge.internal:3000   10.10.20.20:3000       14   195960762      0   23982983
usc1b-edge-aero-20.c.bc-edge.internal:3000   *BB915140A0A0142   usc1b-edge-aero-20.c.bc-edge.internal:3000   10.10.20.21:3000       17   195960762      0   26509640
Number of rows: 20

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Set Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                      Node              Set   Namespace      Set        Mem     Objects     Stop    Disable           Set
                                         .                .           .   Delete       used           .   Writes   Eviction        Enable
                                         .                .           .        .          .           .    Count          .           Xdr
usc1b-edge-aero-01.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.980 M        0   false      use-default
usc1b-edge-aero-02.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.467 M        0   false      use-default
usc1b-edge-aero-03.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     43.569 M        0   false      use-default
usc1b-edge-aero-04.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.724 M        0   false      use-default
usc1b-edge-aero-05.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     43.057 M        0   false      use-default
usc1b-edge-aero-06.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.622 M        0   false      use-default
usc1b-edge-aero-07.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.348 M        0   false      use-default
usc1b-edge-aero-08.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.757 M        0   false      use-default
usc1b-edge-aero-09.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     43.593 M        0   false      use-default
usc1b-edge-aero-10.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     43.178 M        0   false      use-default
usc1b-edge-aero-11.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.623 M        0   false      use-default
usc1b-edge-aero-12.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.800 M        0   false      use-default
usc1b-edge-aero-13.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.354 M        0   false      use-default
usc1b-edge-aero-14.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     43.139 M        0   false      use-default
usc1b-edge-aero-15.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.500 M        0   false      use-default
usc1b-edge-aero-16.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.600 M        0   false      use-default
usc1b-edge-aero-17.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.313 M        0   false      use-default
usc1b-edge-aero-18.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.681 M        0   false      use-default
usc1b-edge-aero-19.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     41.950 M        0   false      use-default
usc1b-edge-aero-20.c.bc-edge.internal:3000   DeviceIPs        BlueCava    false    0.000 B     42.386 M        0   false      use-default
usc1b-edge-aero-01.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.313 M        0   false      use-default
usc1b-edge-aero-02.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.307 M        0   false      use-default
usc1b-edge-aero-03.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.289 M        0   false      use-default
usc1b-edge-aero-04.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.273 M        0   false      use-default
usc1b-edge-aero-05.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.318 M        0   false      use-default
usc1b-edge-aero-06.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.288 M        0   false      use-default
usc1b-edge-aero-07.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.235 M        0   false      use-default
usc1b-edge-aero-08.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.288 M        0   false      use-default
usc1b-edge-aero-09.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.292 M        0   false      use-default
usc1b-edge-aero-10.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.270 M        0   false      use-default
usc1b-edge-aero-11.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.298 M        0   false      use-default
usc1b-edge-aero-12.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.286 M        0   false      use-default
usc1b-edge-aero-13.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.225 M        0   false      use-default
usc1b-edge-aero-14.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.321 M        0   false      use-default
usc1b-edge-aero-15.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.259 M        0   false      use-default
usc1b-edge-aero-16.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.242 M        0   false      use-default
usc1b-edge-aero-17.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.256 M        0   false      use-default
usc1b-edge-aero-18.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.274 M        0   false      use-default
usc1b-edge-aero-19.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.332 M        0   false      use-default
usc1b-edge-aero-20.c.bc-edge.internal:3000   SnapshotHashes   BlueCava    false    0.000 B      5.239 M        0   false      use-default
usc1b-edge-aero-01.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     58.951 M        0   false      use-default
usc1b-edge-aero-02.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     59.432 M        0   false      use-default
usc1b-edge-aero-03.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     60.026 M        0   false      use-default
usc1b-edge-aero-04.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     60.674 M        0   false      use-default
usc1b-edge-aero-05.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     58.932 M        0   false      use-default
usc1b-edge-aero-06.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     59.442 M        0   false      use-default
usc1b-edge-aero-07.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     61.108 M        0   false      use-default
usc1b-edge-aero-08.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     59.815 M        0   false      use-default
usc1b-edge-aero-09.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     60.111 M        0   false      use-default
usc1b-edge-aero-10.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     60.563 M        0   false      use-default
usc1b-edge-aero-11.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     59.770 M        0   false      use-default
usc1b-edge-aero-12.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     59.883 M        0   false      use-default
usc1b-edge-aero-13.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     61.720 M        0   false      use-default
usc1b-edge-aero-14.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     58.896 M        0   false      use-default
usc1b-edge-aero-15.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     60.778 M        0   false      use-default
usc1b-edge-aero-16.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     61.717 M        0   false      use-default
usc1b-edge-aero-17.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     60.630 M        0   false      use-default
usc1b-edge-aero-18.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     60.501 M        0   false      use-default
usc1b-edge-aero-19.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     60.058 M        0   false      use-default
usc1b-edge-aero-20.c.bc-edge.internal:3000   Snapshots        BlueCava    false    0.000 B     61.166 M        0   false      use-default
usc1b-edge-aero-01.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    113.639 M        0   false      use-default
usc1b-edge-aero-02.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    113.876 M        0   false      use-default
usc1b-edge-aero-03.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    112.214 M        0   false      use-default
usc1b-edge-aero-04.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    112.447 M        0   false      use-default
usc1b-edge-aero-05.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    114.093 M        0   false      use-default
usc1b-edge-aero-06.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    113.677 M        0   false      use-default
usc1b-edge-aero-07.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    112.592 M        0   false      use-default
usc1b-edge-aero-08.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    113.505 M        0   false      use-default
usc1b-edge-aero-09.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    112.392 M        0   false      use-default
usc1b-edge-aero-10.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    112.416 M        0   false      use-default
usc1b-edge-aero-11.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    113.737 M        0   false      use-default
usc1b-edge-aero-12.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    113.086 M        0   false      use-default
usc1b-edge-aero-13.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    111.612 M        0   false      use-default
usc1b-edge-aero-14.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    113.935 M        0   false      use-default
usc1b-edge-aero-15.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    112.461 M        0   false      use-default
usc1b-edge-aero-16.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    111.725 M        0   false      use-default
usc1b-edge-aero-17.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    113.033 M        0   false      use-default
usc1b-edge-aero-18.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    112.809 M        0   false      use-default
usc1b-edge-aero-19.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    113.594 M        0   false      use-default
usc1b-edge-aero-20.c.bc-edge.internal:3000   UniqueIds        BlueCava    false    0.000 B    112.297 M        0   false      use-default
Number of rows: 80

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                      Node   Namespace   Avail%   Evictions      Master     Replica     Repl     Stop   Migrates        Disk    Disk     HWM         Mem     Mem    HWM      Stop
                                         .           .        .           .     Objects     Objects   Factor   Writes    (tx,rx)        Used   Used%   Disk%        Used   Used%   Mem%   Writes%
usc1b-edge-aero-01.c.bc-edge.internal:3000   BlueCava        78    28134373   106.614 M   114.270 M        2   false    (0,0)      68.149 GB      18      50   13.166 GB      60     60        90
usc1b-edge-aero-02.c.bc-edge.internal:3000   BlueCava        79    17225300   122.382 M    98.700 M        2   false    (0,0)      68.188 GB      18      50   13.178 GB      60     60        90
usc1b-edge-aero-03.c.bc-edge.internal:3000   BlueCava        76    50723670   103.427 M   117.671 M        2   false    (0,0)      68.701 GB      18      50   13.178 GB      60     60        90
usc1b-edge-aero-04.c.bc-edge.internal:3000   BlueCava        78    31353634   106.325 M   114.794 M        2   false    (0,0)      68.700 GB      18      50   13.180 GB      60     60        90
usc1b-edge-aero-05.c.bc-edge.internal:3000   BlueCava        78    23286506   111.980 M   109.421 M        2   false    (0,0)      68.229 GB      18      50   13.197 GB      60     60        90
usc1b-edge-aero-06.c.bc-edge.internal:3000   BlueCava        79    19690258   113.282 M   107.746 M        2   false    (0,0)      68.214 GB      18      50   13.174 GB      60     60        90
usc1b-edge-aero-07.c.bc-edge.internal:3000   BlueCava        78    22338431   111.791 M   109.492 M        2   false    (0,0)      68.738 GB      18      50   13.190 GB      60     60        90
usc1b-edge-aero-08.c.bc-edge.internal:3000   BlueCava        78    22538324   107.331 M   114.035 M        2   false    (0,0)      68.437 GB      18      50   13.194 GB      60     60        90
usc1b-edge-aero-09.c.bc-edge.internal:3000   BlueCava        76    48860097   101.839 M   119.549 M        2   false    (0,0)      68.793 GB      18      50   13.196 GB      60     60        90
usc1b-edge-aero-10.c.bc-edge.internal:3000   BlueCava        77    40793894   103.707 M   117.719 M        2   false    (0,0)      68.850 GB      18      50   13.198 GB      60     60        90
usc1b-edge-aero-11.c.bc-edge.internal:3000   BlueCava        79    16748394   105.393 M   116.036 M        2   false    (0,0)      68.344 GB      18      50   13.198 GB      60     60        90
usc1b-edge-aero-12.c.bc-edge.internal:3000   BlueCava        78    33824437   115.771 M   105.283 M        2   false    (0,0)      68.439 GB      18      50   13.176 GB      60     60        90
usc1b-edge-aero-13.c.bc-edge.internal:3000   BlueCava        78    32574545   117.237 M   103.674 M        2   false    (0,0)      68.918 GB      18      50   13.167 GB      60     60        90
usc1b-edge-aero-14.c.bc-edge.internal:3000   BlueCava        78    25563849   107.247 M   114.044 M        2   false    (0,0)      68.222 GB      18      50   13.190 GB      60     60        90
usc1b-edge-aero-15.c.bc-edge.internal:3000   BlueCava        78    26391643   108.271 M   112.727 M        2   false    (0,0)      68.651 GB      18      50   13.172 GB      60     60        90
usc1b-edge-aero-16.c.bc-edge.internal:3000   BlueCava        77    36657293   106.093 M   115.192 M        2   false    (0,0)      69.050 GB      18      50   13.190 GB      60     60        90
usc1b-edge-aero-17.c.bc-edge.internal:3000   BlueCava        79    20964129   120.489 M   100.743 M        2   false    (0,0)      68.568 GB      18      50   13.186 GB      60     60        90
usc1b-edge-aero-18.c.bc-edge.internal:3000   BlueCava        78    28724470   114.309 M   106.956 M        2   false    (0,0)      68.650 GB      18      50   13.188 GB      60     60        90
usc1b-edge-aero-19.c.bc-edge.internal:3000   BlueCava        80     6574206   118.374 M   102.561 M        2   false    (0,0)      68.197 GB      18      50   13.169 GB      60     60        90
usc1b-edge-aero-20.c.bc-edge.internal:3000   BlueCava        78    25576247   109.836 M   111.251 M        2   false    (0,0)      68.753 GB      18      50   13.178 GB      60     60        90
Number of rows: 20

qorbani · March 18, 2016, 1:35am

show latency:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~proxy Latency~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                      Node                     Time   Ops/Sec   >1Ms   >8Ms   >64Ms
                                         .                     Span         .      .      .       .
usc1b-edge-aero-01.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       0.0    0.0    0.0     0.0
usc1b-edge-aero-02.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0    0.0    0.0     0.0
usc1b-edge-aero-03.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-04.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0    0.0    0.0     0.0
usc1b-edge-aero-05.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-06.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       0.0    0.0    0.0     0.0
usc1b-edge-aero-07.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0    0.0    0.0     0.0
usc1b-edge-aero-08.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0    0.0    0.0     0.0
usc1b-edge-aero-09.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-10.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0    0.0    0.0     0.0
usc1b-edge-aero-11.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0    0.0    0.0     0.0
usc1b-edge-aero-12.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-13.c.bc-edge.internal:3000   01:32:42-GMT->01:32:52       0.0    0.0    0.0     0.0
usc1b-edge-aero-14.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0    0.0    0.0     0.0
usc1b-edge-aero-15.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0    0.0    0.0     0.0
usc1b-edge-aero-16.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0    0.0    0.0     0.0
usc1b-edge-aero-17.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0    0.0    0.0     0.0
usc1b-edge-aero-18.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-19.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-20.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
Number of rows: 20

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~query Latency~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                      Node                     Time   Ops/Sec   >1Ms   >8Ms   >64Ms
                                         .                     Span         .      .      .       .
usc1b-edge-aero-01.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       0.0    0.0    0.0     0.0
usc1b-edge-aero-02.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0    0.0    0.0     0.0
usc1b-edge-aero-03.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-04.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0    0.0    0.0     0.0
usc1b-edge-aero-05.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-06.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       0.0    0.0    0.0     0.0
usc1b-edge-aero-07.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0    0.0    0.0     0.0
usc1b-edge-aero-08.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0    0.0    0.0     0.0
usc1b-edge-aero-09.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-10.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0    0.0    0.0     0.0
usc1b-edge-aero-11.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0    0.0    0.0     0.0
usc1b-edge-aero-12.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-13.c.bc-edge.internal:3000   01:32:42-GMT->01:32:52       0.0    0.0    0.0     0.0
usc1b-edge-aero-14.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0    0.0    0.0     0.0
usc1b-edge-aero-15.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0    0.0    0.0     0.0
usc1b-edge-aero-16.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0    0.0    0.0     0.0
usc1b-edge-aero-17.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0    0.0    0.0     0.0
usc1b-edge-aero-18.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-19.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-20.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
Number of rows: 20

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~reads Latency~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                      Node                     Time   Ops/Sec   >1Ms   >8Ms   >64Ms
                                         .                     Span         .      .      .       .
usc1b-edge-aero-01.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       0.0    0.0    0.0     0.0
usc1b-edge-aero-02.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0    0.0    0.0     0.0
usc1b-edge-aero-03.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-04.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0    0.0    0.0     0.0
usc1b-edge-aero-05.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-06.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       0.0    0.0    0.0     0.0
usc1b-edge-aero-07.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0    0.0    0.0     0.0
usc1b-edge-aero-08.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0    0.0    0.0     0.0
usc1b-edge-aero-09.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-10.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0    0.0    0.0     0.0
usc1b-edge-aero-11.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0    0.0    0.0     0.0
usc1b-edge-aero-12.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-13.c.bc-edge.internal:3000   01:32:42-GMT->01:32:52       0.0    0.0    0.0     0.0
usc1b-edge-aero-14.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0    0.0    0.0     0.0
usc1b-edge-aero-15.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0    0.0    0.0     0.0
usc1b-edge-aero-16.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0    0.0    0.0     0.0
usc1b-edge-aero-17.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0    0.0    0.0     0.0
usc1b-edge-aero-18.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-19.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-20.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
Number of rows: 20

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~udf Latency~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                      Node                     Time   Ops/Sec   >1Ms   >8Ms   >64Ms
                                         .                     Span         .      .      .       .
usc1b-edge-aero-01.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       0.0    0.0    0.0     0.0
usc1b-edge-aero-02.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0    0.0    0.0     0.0
usc1b-edge-aero-03.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-04.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0    0.0    0.0     0.0
usc1b-edge-aero-05.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-06.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       0.0    0.0    0.0     0.0
usc1b-edge-aero-07.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0    0.0    0.0     0.0
usc1b-edge-aero-08.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0    0.0    0.0     0.0
usc1b-edge-aero-09.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-10.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0    0.0    0.0     0.0
usc1b-edge-aero-11.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0    0.0    0.0     0.0
usc1b-edge-aero-12.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-13.c.bc-edge.internal:3000   01:32:42-GMT->01:32:52       0.0    0.0    0.0     0.0
usc1b-edge-aero-14.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0    0.0    0.0     0.0
usc1b-edge-aero-15.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0    0.0    0.0     0.0
usc1b-edge-aero-16.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0    0.0    0.0     0.0
usc1b-edge-aero-17.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0    0.0    0.0     0.0
usc1b-edge-aero-18.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
usc1b-edge-aero-19.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0    0.0    0.0     0.0
usc1b-edge-aero-20.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0    0.0    0.0     0.0
Number of rows: 20

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~writes_master Latency~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                      Node                     Time   Ops/Sec    >1Ms    >8Ms   >64Ms
                                         .                     Span         .       .       .       .
usc1b-edge-aero-01.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       3.3   100.0   100.0   100.0
usc1b-edge-aero-02.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0     0.0     0.0     0.0
usc1b-edge-aero-03.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0     0.0     0.0     0.0
usc1b-edge-aero-04.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0     0.0     0.0     0.0
usc1b-edge-aero-05.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0     0.0     0.0     0.0
usc1b-edge-aero-06.c.bc-edge.internal:3000   01:32:41-GMT->01:32:51       0.0     0.0     0.0     0.0
usc1b-edge-aero-07.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0     0.0     0.0     0.0
usc1b-edge-aero-08.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0     0.0     0.0     0.0
usc1b-edge-aero-09.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0     0.0     0.0     0.0
usc1b-edge-aero-10.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0     0.0     0.0     0.0
usc1b-edge-aero-11.c.bc-edge.internal:3000   01:32:44-GMT->01:32:54       0.0     0.0     0.0     0.0
usc1b-edge-aero-12.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56      29.4   36.05   35.71   35.71
usc1b-edge-aero-13.c.bc-edge.internal:3000   01:32:42-GMT->01:32:52       1.5   53.33   46.67   46.67
usc1b-edge-aero-14.c.bc-edge.internal:3000   01:32:39-GMT->01:32:49       0.0     0.0     0.0     0.0
usc1b-edge-aero-15.c.bc-edge.internal:3000   01:32:47-GMT->01:32:57       0.0     0.0     0.0     0.0
usc1b-edge-aero-16.c.bc-edge.internal:3000   01:32:45-GMT->01:32:55       0.0     0.0     0.0     0.0
usc1b-edge-aero-17.c.bc-edge.internal:3000   01:32:38-GMT->01:32:48       0.0     0.0     0.0     0.0
usc1b-edge-aero-18.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       4.7   40.43   40.43   40.43
usc1b-edge-aero-19.c.bc-edge.internal:3000   01:32:40-GMT->01:32:50       0.0     0.0     0.0     0.0
usc1b-edge-aero-20.c.bc-edge.internal:3000   01:32:46-GMT->01:32:56       0.0     0.0     0.0     0.0
Number of rows: 20

anushree · March 18, 2016, 11:09pm

Looks like some of the nodes are getting hit with high latencies, and yes the others are not accepting any operations/second.

I think we are going to need to enable micro-benchmarks.

Once that is enabled, you use asloglatency to identify which components seem to be creating the lag on the nodes.

I assume the restore is being done remotely? Which node amongst the cluster have you pointed it towards?

Topic		Replies	Views
Asbackup failing asbackup & asrestore	3	3598	February 15, 2017
How to speed up asrestore speed? asbackup & asrestore	2	2175	July 18, 2018
Error while storing record - code 4 asbackup & asrestore	1	387	July 4, 2023
Asbackup takes too long asbackup & asrestore	6	2878	February 25, 2016
Asrestore problem: put failed in restore: unusual error 18 trying again asbackup & asrestore	4	3508	August 26, 2015

Increase Timeout for asrestore

Related topics