Very strange Aerospike behaviour - completely unknown cause

Hi all

Firstly I’m not even sure where the best place to post this is, so I’ve put it in here as it’s kinda connected to the PHP Client library (in that it’s something running PHP) but I don’t even know if this is the cause or effect at this stage. I’m tagging @rbotzer and @kporter because I think this may require everyone’s skills ha ha.

Okay, so back story… we’re running Aerospike CE and have been quite successfully for some time now (couple of years) and everything has been hunky dory - however about 3 weeks ago, we updated our configs to make the memory larger and disk space smaller as our use case changed slightly. Once we restarted Aerospike, we ran into a problem about 10 minutes later of large connections and apps struggling with timeouts etc. Thinking this was related to the updates, we reverted the configs and restarted - but the same thing happened and we were noticing strange “sawtooth” patterns in New Relic of the apps accessing aerospike. Connections would steady out at around 800 per node, then every 10 minutes increase to over 5k before slowly coming back down to ~800 again. We brought the cluster down, updated configs for the desired end results (more ram, less disk usage), zeroed out the drives as we had the data and then restarted the cluster - still the same issue occurred.

Now, as an aside, we currently have a CRON job which takes data from MySQL, formats it and syncs it into Aerospike - this runs every 10 minutes and so we naturally thought there was a connection. We stopped the CRON job, and this only re-created the issue with the rising connections etc so instead we reduced the time to 5 minutes and this seemed to alleviate the problem with connections. However, we still see the “sawtooth” pattern in New Relic in terms of response time from Aerospike, but at least this meant the service remained stable.

This brings me on to the main issue we are having now. If we stop the CRON running the sync to Aerospike, at almost exactly the 10 minute mark, the connection rates increase massively (up to ~5k) and the whole system grinds to a halt, the apps time out, connections can’t be made etc and the whole platform will eventually die. If we start the CRON again and run a sync, the connections slowly drop and everything carries on as normal - which is why this is the strangest thing I’ve ever seen. I’m basically asking for your help in being able to debug how turning OFF a CRON job accessing Aerospike actually breaks aerospike, almost in 10 minutes cycles. I understand this is almost completely “I have no idea without looking” kind of scenario, but it’s the “looking” bit we need help with.

I’ve upped the log level, checked it and can’t immediately see anything obvious, no errors or anything. Does aerospike do anything in 10 minute cycles like GC etc which could cause this (and is reset by something accessing Aerospike)? We’ve updated to the latest version of both client (3.4.14) and server (3.4.1.3) now and still the same The cron accesses via PHP-CLI and the app servers are running PHP-FPM

I really hope you guys can help me in tracking down possible errors or symptoms, really appreciate the support.

Craig

Okay, so here’s an update - I’m guessing it may be something to do with querying and possibly that the cron writing data back to aerospike is somehow either resetting the sindex or the data or some form of query counters or something?

I have run a histogram on the query for that namespace as below, and as you can see I turned off the cron, and at exactly 14:22 you can see the queries start to slow down and connections increase until it gets to a point where we start getting timeouts and then I re-enable the cron at 14:26:56, run a sync and things start to settle back down again:

Sep 14 2017 14:17:26
               % > (ms)
slice-to (sec)      1      8     64    ops/sec
-------------- ------ ------ ------ ----------
14:17:36    10  29.68   0.83   0.01     1011.2
14:17:46    10  30.02   0.57   0.00     1023.4
14:17:56    10  29.11   0.54   0.00     1016.1
14:18:06    10  29.59   0.40   0.00     1027.8
14:18:16    10  30.21   0.25   0.00      999.4
14:18:26    10  27.03   0.51   0.00      988.8
14:18:36    10  30.09   0.33   0.00      990.8
14:18:46    10  29.77   0.20   0.00     1028.8
14:18:56    10  29.67   0.51   0.00     1012.3
14:19:06    10  28.69   0.73   0.00     1039.3
14:19:16    10  29.76   0.56   0.00      981.9
14:19:26    10  27.55   0.86   0.00     1011.9
14:19:36    10  27.52   1.55   0.02     1031.6
14:19:46    10  27.55   1.70   0.01     1050.6
14:19:56    10  27.66   0.73   0.00     1016.8
14:20:06    10  27.22   0.47   0.00     1024.1
14:20:16    10  27.08   0.65   0.00     1054.0
14:20:26    10  25.97   0.46   0.00     1029.0
14:20:36    10  28.20   0.57   0.00     1016.9
14:20:46    10  29.47   0.90   0.00      989.9
14:20:56    10  30.69   0.56   0.00      985.1
14:21:06    10  29.34   0.60   0.00     1016.7
14:21:16    10  30.23   0.61   0.00      989.8
14:21:26    10  29.90   0.39   0.00      990.0
14:21:36    10  29.58   0.45   0.00     1008.0
14:21:46    10  29.86   0.54   0.00     1016.1
14:21:56    10  31.49   0.79   0.00     1019.6
14:22:06    10  49.85  16.46   0.00     1032.9
14:22:16    10  57.00  18.56   0.00     1004.6
14:22:26    10  65.38  20.42   0.00     1029.7
14:22:36    10  68.66  19.89   0.00      994.3
14:22:46    10  71.37  17.52   0.00      997.3
14:22:56    10  71.43  17.58   0.00      993.5
14:23:06    10  71.74  16.54   0.00     1027.1
14:23:16    10  71.27  14.89   0.00     1031.2
14:23:26    10  71.30  25.44   0.01     1013.0
14:23:36    10  71.82  62.48  23.32      927.4
14:23:46    10  71.72  63.57  24.73      915.9
14:23:56    10  72.38  68.15  43.12      905.4
14:24:06    10  71.87  66.89  37.14      915.0
14:24:16    10  71.54  63.94  21.96      880.4
14:24:26    10  71.59  61.68  37.51      843.6
14:24:36    10  71.52  68.19  53.40      906.1
14:24:46    10  71.71  71.07  71.07      941.8
14:24:56    10  72.16  64.21  40.34      856.9
14:25:06    10  70.78  59.69  32.06      803.5
14:25:16    10  70.81  61.21  50.98      846.1
14:25:26    10  72.57  67.73  54.83      876.3
14:25:36    10  71.68  60.90  42.47      853.5
14:25:46    10  70.50  56.07  40.48      909.6
14:25:56    10  72.00  59.38  47.10      854.7
14:26:06    10  72.64  66.00  53.21      891.3
14:26:16    10  71.65  64.29  53.48      849.6
14:26:26    10  72.21  62.42  44.23      832.6
14:26:36    10  71.36  62.47  47.23      813.5
14:26:46    10  71.34  64.92  51.24      853.4
14:26:56    10  69.34  35.80  17.72     1151.1
14:27:06    10  72.89  21.81  17.61     1511.4
14:27:16    10  71.96  20.17  16.51     1332.9
14:27:26    10  71.89  17.70  11.29     1199.7
14:27:36    10  71.18  11.36   1.52     1095.9
14:27:46    10  71.15   3.00   0.00     1044.1
14:27:56    10  71.09   3.46   0.00     1051.7
14:28:06    10  70.86   2.99   0.01     1026.0
14:28:16    10  70.65   2.01   0.00     1010.0
14:28:26    10  70.67   2.17   0.00     1021.4
14:28:36    10  70.25   2.03   0.00      987.1
14:28:46    10  70.56   0.97   0.00     1011.1
14:28:56    10  70.30   0.89   0.00      993.4
14:29:06    10  69.83   0.89   0.00     1009.3
14:29:16    10  69.44   0.94   0.00     1037.2
14:29:26    10  68.91   0.54   0.00     1001.3
14:29:36    10  69.48   0.77   0.00     1042.4
14:29:46    10  69.00   0.63   0.00     1013.9
14:29:56    10  69.44   0.48   0.00     1043.7
14:30:06    10  69.40   1.60   0.00      953.8
14:30:16    10  69.11   1.04   0.00     1041.3
-------------- ------ ------ ------ ----------
     avg        55.74  20.66  12.14      994.0
     max        72.89  71.07  71.07     1511.4

Could there be something that builds up, some kind of limit we’re hitting with queries etc? How would we test and find out?

Thanks

  1. Could you provide your config?

  2. Could be overwhelming you storage layer? Assuming sysstat is installed, could you run the following:

    sar -d
    

Could try query microbenchmarks to help identify the bottleneck there. Is the latency across all operations? Do you only use query? Aerospike Technical Documentation | Aerospike Documentation

Hi @Albot - I ran microbenchmarks for querying and posted the details above as I wanted to see if this was the issue. As you can see it’s mainly queries which slow down. To give more context…

We have 2 main applications which interface with Aerospike, 1 does reads, writes and queries and the other simply does reads / writes.

When everything goes pear shaped, the application doing the queries is the one which is affected, not the one doing just reads / writes (apart from a slight increase in response times, but hardly noticeable) - this is what leads me to think it’s query based - maybe even sindex based?

I guess my lack of understanding is how the running cron (which basically writes the data the queries use) somehow “resets” something to allow the queries to continue running and not mount up? I have also checked in our monitoring and stats system (CheckMK) and when everything grinds down, the CPU usage shows an increasing amount of time spent in IO wait, so this may indeed be what @kporter is referring to. I’ll post an example now (also shows the 5 minute saw tooth)…

image

The 2 spikes are where I have disabled the CRON so it’s not writing the data to Aerospike

Sure, here is the full config (I’m guessing you don’t mean the actual config file):

paxos-single-replica-limit=1
pidfile=/var/run/aerospike/asd.pid
proto-fd-max=70000
advertise-ipv6=false
auto-pin=none
batch-threads=4
batch-max-buffers-per-queue=255
batch-max-requests=5000
batch-max-unused-buffers=256
batch-priority=200
batch-index-threads=12
clock-skew-max-ms=1000
cluster-name=null
enable-benchmarks-fabric=false
enable-benchmarks-svc=false
enable-hist-info=false
hist-track-back=300
hist-track-slice=10
hist-track-thresholds=null
info-threads=16
ldt-benchmarks=false
log-local-time=false
migrate-max-num-incoming=4
migrate-threads=1
min-cluster-size=1
node-id-interface=null
nsup-delete-sleep=100
nsup-period=120
nsup-startup-evict=true
proto-fd-idle-ms=60000
proto-slow-netio-sleep-ms=1
query-batch-size=100
query-buf-size=2097152
query-bufpool-size=256
query-in-transaction-thread=false
query-long-q-max-size=500
query-microbenchmark=false
query-pre-reserve-partitions=false
query-priority=10
query-priority-sleep-us=1
query-rec-count-bound=18446744073709551615
query-req-in-query-thread=false
query-req-max-inflight=100
query-short-q-max-size=500
query-threads=6
query-threshold=10
query-untracked-time-ms=1000
query-worker-threads=15
respond-client-on-master-completion=false
run-as-daemon=true
scan-max-active=100
scan-max-done=100
scan-max-udf-transactions=32
scan-threads=4
service-threads=4
sindex-builder-threads=4
sindex-gc-max-rate=50000
sindex-gc-period=10
ticker-interval=10
transaction-max-ms=1000
transaction-pending-limit=20
transaction-queues=4
transaction-repeatable-read=false
transaction-retry-ms=1002
transaction-threads-per-queue=4
work-directory=/opt/aerospike
write-duplicate-resolution-disable=false
debug-allocations=none
fabric-dump-msgs=false
max-msgs-per-type=-1
prole-extra-ttl=0
service.port=3000
service.address=any
service.access-port=0
service.access-address=192.168.120.51
service.alternate-access-port=0
service.tls-port=0
service.tls-access-port=0
service.tls-alternate-access-port=0
service.tls-name=null
heartbeat.mode=multicast
heartbeat.address=192.168.60.51
heartbeat.multicast-group=239.1.99.51
heartbeat.port=9918
heartbeat.interval=150
heartbeat.timeout=10
heartbeat.mtu=1500
heartbeat.protocol=v3
fabric.address=any
fabric.port=3001
fabric.channel-bulk-fds=2
fabric.channel-bulk-recv-threads=4
fabric.channel-ctrl-fds=1
fabric.channel-ctrl-recv-threads=4
fabric.channel-meta-fds=1
fabric.channel-meta-recv-threads=4
fabric.channel-rw-fds=8
fabric.channel-rw-recv-threads=16
fabric.keepalive-enabled=true
fabric.keepalive-intvl=1
fabric.keepalive-probes=10
fabric.keepalive-time=1
fabric.latency-max-ms=5
fabric.recv-rearm-threshold=1024
fabric.send-threads=8
info.address=any
info.port=3003
enable-security=false
privilege-refresh-period=300
report-authentication-sinks=0
report-data-op-sinks=0
report-sys-admin-sinks=0
report-user-admin-sinks=0
report-violation-sinks=0
syslog-local=-1

If there’s anything else you need let me know - we’re running a 2 node cluster, with 2 namespaces - 1 “permanent” and 1 “temp” which has a default TTL of 24 hours.

this is the output of sar -d (although bear in mind that the cron is still running and so we’re not seeing the slow down currently or in this time range - would it help to run this when everything is grinding?)

00:00:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
00:10:01       dev8-0      0.23      0.00      3.52     15.29      0.00      0.00      0.00      0.00
00:10:01      dev8-16    128.17   2859.11   2715.08     43.49      0.02      0.19      0.19      2.40
00:10:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
00:10:01      dev8-48      1.08      1.28    276.46    256.00      0.00      0.32      0.32      0.03
00:10:01      dev8-32      1.10      1.71    279.44    256.00      0.00      0.32      0.32      0.04
00:10:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
00:10:01      dev8-64      1.08      0.85    276.88    256.00      0.00      0.32      0.32      0.03
00:10:01     dev253-0      0.24      0.00      3.52     14.65      0.00      0.00      0.00      0.00
00:10:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:10:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:20:01       dev8-0      0.18      0.00      2.78     15.59      0.00      0.00      0.00      0.00
00:20:01      dev8-16    135.85   2993.39   2838.95     42.93      0.03      0.20      0.19      2.60
00:20:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
00:20:01      dev8-48      1.10      1.71    279.03    256.00      0.00      0.34      0.34      0.04
00:20:01      dev8-32      1.11      2.56    280.74    256.00      0.00      0.35      0.35      0.04
00:20:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
00:20:01      dev8-64      1.09      0.43    279.03    256.00      0.00      0.33      0.33      0.04
00:20:01     dev253-0      0.18      0.00      2.78     15.30      0.00      0.00      0.00      0.00
00:20:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:20:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:30:01       dev8-0      0.18      0.00      2.98     16.69      0.00      0.00      0.00      0.00
00:30:01      dev8-16    141.41   3076.08   2870.09     42.05      0.03      0.19      0.18      2.58
00:30:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
00:30:01      dev8-48      1.10      1.71    279.03    256.00      0.00      0.34      0.34      0.04
00:30:01      dev8-32      1.09      1.71    278.18    256.00      0.00      0.33      0.32      0.04
00:30:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
00:30:01      dev8-64      1.09      2.13    276.90    256.00      0.00      0.31      0.31      0.03
00:30:01     dev253-0      0.18      0.00      2.98     16.09      0.00      0.00      0.00      0.00
00:30:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:30:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:40:01       dev8-0      0.16      0.00      2.75     17.39      0.00      0.00      0.00      0.00
00:40:01      dev8-16    146.55   3120.32   2923.80     41.24      0.03      0.19      0.18      2.68
00:40:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
00:40:01      dev8-48      1.10      2.99    279.45    256.00      0.00      0.37      0.37      0.04
00:40:01      dev8-32      1.11      2.99    280.73    256.00      0.00      0.36      0.36      0.04
00:40:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
00:40:01      dev8-64      1.09      1.71    276.89    256.00      0.00      0.33      0.33      0.04
00:40:01     dev253-0      0.17      0.00      2.75     16.20      0.00      0.00      0.00      0.00
00:40:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:40:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:50:01       dev8-0      0.15      0.00      2.74     18.08      0.00      0.00      0.00      0.00
00:50:01      dev8-16    157.67   3210.31   3007.90     39.44      0.03      0.19      0.18      2.84
00:50:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
00:50:01      dev8-48      1.09      1.71    278.18    256.00      0.00      0.32      0.32      0.03
00:50:01      dev8-32      1.11      2.99    282.02    256.00      0.00      0.34      0.33      0.04
00:50:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
00:50:01      dev8-64      1.09      1.28    276.90    256.00      0.00      0.32      0.31      0.03
00:50:01     dev253-0      0.16      0.00      2.74     17.14      0.00      0.00      0.00      0.00
00:50:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:50:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

00:50:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
01:00:01       dev8-0      0.17      0.00      2.89     17.36      0.00      0.00      0.00      0.00
01:00:01      dev8-16    166.95   3332.95   3101.71     38.54      0.03      0.19      0.18      3.09
01:00:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
01:00:01      dev8-48      1.09      1.28    278.17    256.00      0.00      0.35      0.35      0.04
01:00:01      dev8-32      1.10      1.71    279.45    256.00      0.00      0.35      0.34      0.04
01:00:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
01:00:01      dev8-64      1.08      0.85    276.47    256.00      0.00      0.36      0.36      0.04
01:00:01     dev253-0      0.18      0.00      2.89     16.22      0.00      0.00      0.00      0.00
01:00:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:00:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:10:01       dev8-0      0.22      0.00      3.31     15.04      0.00      0.00      0.00      0.00
01:10:01      dev8-16    178.71   3427.16   3165.66     36.89      0.03      0.19      0.18      3.16
01:10:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
01:10:01      dev8-48      1.09      2.13    278.17    256.00      0.00      0.33      0.33      0.04
01:10:01      dev8-32      1.09      0.43    278.59    256.00      0.00      0.31      0.31      0.03
01:10:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
01:10:01      dev8-64      1.09      2.13    277.74    256.00      0.00      0.31      0.31      0.03
01:10:01     dev253-0      0.23      0.00      3.31     14.38      0.00      0.00      0.00      0.00
01:10:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:10:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:01       dev8-0      0.18      0.00      2.84     15.93      0.00      0.00      0.00      0.00
01:20:01      dev8-16    188.24   3588.25   3273.65     36.45      0.03      0.18      0.18      3.33
01:20:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.05      0.05      0.00
01:20:01      dev8-48      1.09      1.71    278.17    256.00      0.00      0.33      0.33      0.04
01:20:01      dev8-32      1.10      2.56    280.31    256.00      0.00      0.35      0.35      0.04
01:20:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
01:20:01      dev8-64      1.08      0.85    276.47    256.00      0.00      0.34      0.34      0.04
01:20:01     dev253-0      0.18      0.00      2.84     15.79      0.00      0.00      0.00      0.00
01:20:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:30:01       dev8-0      0.22      0.00      3.48     15.45      0.00      0.01      0.01      0.00
01:30:01      dev8-16    197.36   3647.28   3359.89     35.51      0.03      0.18      0.17      3.37
01:30:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
01:30:01      dev8-48      1.09      0.00    278.18    256.00      0.00      0.29      0.29      0.03
01:30:01      dev8-32      1.11      2.13    281.16    256.00      0.00      0.33      0.33      0.04
01:30:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
01:30:01      dev8-64      1.09      0.43    278.60    256.00      0.00      0.31      0.31      0.03
01:30:01     dev253-0      0.23      0.00      3.48     15.12      0.00      0.01      0.01      0.00
01:30:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:30:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:40:01       dev8-0      0.23      0.00      3.62     15.53      0.00      0.00      0.00      0.00
01:40:01      dev8-16    223.43   3851.21   3535.24     33.06      0.04      0.18      0.17      3.76
01:40:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
01:40:01      dev8-48      1.10      2.99    279.03    256.00      0.00      0.35      0.35      0.04
01:40:01      dev8-32      1.10      2.13    279.88    256.00      0.00      0.31      0.31      0.03
01:40:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
01:40:01      dev8-64      1.09      1.71    276.90    256.00      0.00      0.34      0.34      0.04
01:40:01     dev253-0      0.25      0.00      3.62     14.40      0.00      0.00      0.00      0.00
01:40:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:40:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

01:40:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
01:50:01       dev8-0      0.18      0.00      2.81     15.75      0.00      0.00      0.00      0.00
01:50:01      dev8-16    236.71   4051.28   3677.32     32.65      0.04      0.17      0.16      3.85
01:50:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.01      0.01      0.00
01:50:01      dev8-48      1.10      1.71    279.03    256.00      0.00      0.31      0.31      0.03
01:50:01      dev8-32      1.11      2.13    281.59    256.00      0.00      0.33      0.33      0.04
01:50:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
01:50:01      dev8-64      1.09      0.43    279.03    256.00      0.00      0.32      0.32      0.03
01:50:01     dev253-0      0.18      0.00      2.81     15.60      0.00      0.00      0.00      0.00
01:50:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:50:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:00:01       dev8-0      0.20      0.00      3.21     15.80      0.00      0.00      0.00      0.00
02:00:01      dev8-16    255.10   4185.62   3792.88     31.28      0.04      0.17      0.16      4.21
02:00:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
02:00:01      dev8-48      1.10      2.13    279.45    256.00      0.00      0.33      0.33      0.04
02:00:01      dev8-32      1.10      2.13    280.73    256.00      0.00      0.35      0.34      0.04
02:00:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
02:00:01      dev8-64      1.10      1.71    279.03    256.00      0.00      0.33      0.33      0.04
02:00:01     dev253-0      0.21      0.00      3.21     15.18      0.00      0.00      0.00      0.00
02:00:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:00:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01       dev8-0      0.23      0.00      3.47     15.19      0.00      0.00      0.00      0.00
02:10:01      dev8-16    260.16   4349.83   3899.84     31.71      0.04      0.17      0.16      4.23
02:10:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
02:10:01      dev8-48      1.08      0.00    276.88    256.00      0.00      0.34      0.33      0.04
02:10:01      dev8-32      1.10      1.71    279.87    256.00      0.00      0.32      0.32      0.03
02:10:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
02:10:01      dev8-64      1.09      1.28    277.74    256.00      0.00      0.33      0.33      0.04
02:10:01     dev253-0      0.24      0.00      3.47     14.45      0.00      0.00      0.00      0.00
02:10:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:20:01       dev8-0      0.18      0.00      2.89     16.20      0.00      0.00      0.00      0.00
02:20:01      dev8-16    263.88   4390.04   3912.40     31.46      0.05      0.17      0.17      4.38
02:20:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
02:20:01      dev8-48      1.10      2.56    279.88    256.00      0.00      0.32      0.31      0.03
02:20:01      dev8-32      1.11      2.56    282.02    256.00      0.00      0.34      0.34      0.04
02:20:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
02:20:01      dev8-64      1.09      1.28    277.32    256.00      0.00      0.32      0.32      0.03
02:20:01     dev253-0      0.18      0.00      2.89     15.90      0.00      0.00      0.00      0.00
02:20:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:20:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:30:01       dev8-0      0.20      0.00      3.19     15.55      0.00      0.00      0.00      0.00
02:30:01      dev8-16    289.71   4533.86   4040.83     29.60      0.05      0.16      0.16      4.53
02:30:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
02:30:01      dev8-48      1.09      1.28    279.03    256.00      0.00      0.32      0.32      0.03
02:30:01      dev8-32      1.11      2.13    281.59    256.00      0.00      0.32      0.32      0.04
02:30:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
02:30:01      dev8-64      1.09      1.28    277.75    256.00      0.00      0.31      0.31      0.03
02:30:01     dev253-0      0.21      0.00      3.19     14.95      0.00      0.00      0.00      0.00
02:30:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:30:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

02:30:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
02:40:01       dev8-0      0.21      0.00      3.29     15.43      0.00      0.00      0.00      0.00
02:40:01      dev8-16    270.36   4665.17   4036.49     32.19      0.05      0.17      0.17      4.49
02:40:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.05      0.04      0.00
02:40:01      dev8-48      1.09      1.28    278.17    256.00      0.00      0.31      0.31      0.03
02:40:01      dev8-32      1.10      1.71    280.73    256.00      0.00      0.35      0.35      0.04
02:40:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
02:40:01      dev8-64      1.10      1.28    279.45    256.00      0.00      0.32      0.32      0.03
02:40:01     dev253-0      0.22      0.00      3.29     14.74      0.00      0.00      0.00      0.00
02:40:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:40:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:50:01       dev8-0      0.18      0.00      2.81     15.57      0.00      0.00      0.00      0.00
02:50:01      dev8-16    195.02   4379.67   3846.71     42.18      0.04      0.19      0.18      3.57
02:50:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.01      0.01      0.00
02:50:01      dev8-48      1.10      1.28    279.92    256.00      0.00      0.36      0.36      0.04
02:50:01      dev8-32      1.12      2.56    282.92    256.00      0.00      0.36      0.36      0.04
02:50:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
02:50:01      dev8-64      1.09      0.85    277.79    256.00      0.00      0.32      0.32      0.03
02:50:01     dev253-0      0.18      0.00      2.81     15.29      0.00      0.00      0.00      0.00
02:50:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:50:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:00:01       dev8-0      0.20      0.00      3.23     15.76      0.00      0.00      0.00      0.00
03:00:01      dev8-16    191.33   3804.14   3555.24     38.46      0.04      0.19      0.18      3.41
03:00:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
03:00:01      dev8-48      1.10      1.28    279.45    256.00      0.00      0.32      0.32      0.03
03:00:01      dev8-32      1.11      2.13    283.29    256.00      0.00      0.35      0.35      0.04
03:00:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
03:00:01      dev8-64      1.10      1.71    280.31    256.00      0.00      0.33      0.33      0.04
03:00:01     dev253-0      0.21      0.00      3.23     15.03      0.00      0.00      0.00      0.00
03:00:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:00:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:10:01       dev8-0      0.28      0.00      4.30     15.56      0.00      0.01      0.01      0.00
03:10:01      dev8-16    187.17   3840.09   3558.53     39.53      0.03      0.18      0.18      3.34
03:10:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
03:10:01      dev8-48      1.11      2.56    280.72    256.00      0.00      0.33      0.33      0.04
03:10:01      dev8-32      1.11      2.13    282.00    256.00      0.00      0.36      0.36      0.04
03:10:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
03:10:01      dev8-64      1.09      1.28    279.02    256.00      0.00      0.32      0.32      0.04
03:10:01     dev253-0      0.30      0.00      4.30     14.27      0.00      0.01      0.01      0.00
03:10:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:10:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:20:01       dev8-0      0.18      0.00      2.89     16.20      0.00      0.00      0.00      0.00
03:20:01      dev8-16    186.54   3791.71   3542.50     39.32      0.03      0.19      0.18      3.37
03:20:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
03:20:01      dev8-48      1.10      1.71    279.88    256.00      0.00      0.33      0.33      0.04
03:20:01      dev8-32      1.11      2.13    281.16    256.00      0.00      0.34      0.34      0.04
03:20:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.03      0.00
03:20:01      dev8-64      1.08      0.43    276.90    256.00      0.00      0.33      0.33      0.04
03:20:01     dev253-0      0.18      0.00      2.89     15.90      0.00      0.00      0.00      0.00
03:20:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:20:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

03:20:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
03:30:01       dev8-0      0.20      0.00      3.13     15.51      0.00      0.00      0.00      0.00
03:30:01      dev8-16    191.75   3904.38   3611.19     39.19      0.03      0.18      0.18      3.37
03:30:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
03:30:01      dev8-48      1.09      1.71    278.60    256.00      0.00      0.38      0.38      0.04
03:30:01      dev8-32      1.11      2.13    281.16    256.00      0.00      0.34      0.34      0.04
03:30:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
03:30:01      dev8-64      1.09      1.71    276.90    256.00      0.00      0.35      0.35      0.04
03:30:01     dev253-0      0.21      0.00      3.13     15.02      0.00      0.00      0.00      0.00
03:30:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:30:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:40:01       dev8-0      0.17      0.00      2.91     16.80      0.00      0.00      0.00      0.00
03:40:01      dev8-16    188.79   3902.31   3592.41     39.70      0.03      0.19      0.18      3.39
03:40:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
03:40:01      dev8-48      1.09      0.85    279.46    256.00      0.00      0.31      0.31      0.03
03:40:01      dev8-32      1.11      1.71    281.59    256.00      0.00      0.34      0.34      0.04
03:40:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
03:40:01      dev8-64      1.09      0.85    279.03    256.00      0.00      0.31      0.31      0.03
03:40:01     dev253-0      0.18      0.00      2.91     16.33      0.00      0.00      0.00      0.00
03:40:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:40:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:50:01       dev8-0      0.46      0.00     13.66     29.58      0.00      0.06      0.03      0.00
03:50:01      dev8-16    189.63   3967.56   3628.68     40.06      0.04      0.19      0.18      3.41
03:50:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
03:50:01      dev8-48      1.11      2.56    280.74    256.00      0.00      0.36      0.36      0.04
03:50:01      dev8-32      1.11      2.13    281.16    256.00      0.00      0.36      0.35      0.04
03:50:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.05      0.05      0.00
03:50:01      dev8-64      1.09      1.71    278.60    256.00      0.00      0.36      0.36      0.04
03:50:01     dev253-0      0.58      0.00     13.66     23.35      0.00      0.08      0.02      0.00
03:50:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:50:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:00:01       dev8-0      0.16      0.00      2.93     18.52      0.00      0.00      0.00      0.00
04:00:01      dev8-16    195.90   3999.83   3649.59     39.05      0.04      0.21      0.18      3.50
04:00:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
04:00:01      dev8-48      1.09      0.85    278.60    256.00      0.00      0.34      0.33      0.04
04:00:01      dev8-32      1.10      1.28    281.16    256.00      0.00      0.33      0.33      0.04
04:00:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
04:00:01      dev8-64      1.09      0.43    278.18    256.00      0.00      0.33      0.32      0.04
04:00:01     dev253-0      0.16      0.00      2.93     17.77      0.00      0.00      0.00      0.00
04:00:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:00:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:01       dev8-0      0.22      0.00      3.41     15.63      0.00      0.00      0.00      0.00
04:10:01      dev8-16    192.60   3974.70   3605.52     39.36      0.03      0.18      0.17      3.35
04:10:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
04:10:01      dev8-48      1.11      2.13    281.15    256.00      0.00      0.33      0.33      0.04
04:10:01      dev8-32      1.11      1.28    282.01    256.00      0.00      0.35      0.35      0.04
04:10:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
04:10:01      dev8-64      1.10      2.13    280.73    256.00      0.00      0.34      0.34      0.04
04:10:01     dev253-0      0.23      0.00      3.41     14.52      0.00      0.00      0.00      0.00
04:10:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

04:10:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
04:20:01       dev8-0      0.18      0.00      2.90     15.96      0.00      0.00      0.00      0.00
04:20:01      dev8-16    187.39   3898.68   3551.82     39.76      0.03      0.18      0.18      3.30
04:20:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
04:20:01      dev8-48      1.11      2.99    281.16    256.00      0.00      0.32      0.31      0.03
04:20:01      dev8-32      1.11      2.13    281.16    256.00      0.00      0.34      0.33      0.04
04:20:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
04:20:01      dev8-64      1.10      1.71    279.45    256.00      0.00      0.34      0.33      0.04
04:20:01     dev253-0      0.19      0.00      2.90     15.54      0.00      0.00      0.00      0.00
04:20:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:20:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:01       dev8-0      0.20      0.00      3.15     15.48      0.00      0.00      0.00      0.00
04:30:01      dev8-16    198.90   3945.19   3627.46     38.07      0.04      0.18      0.17      3.46
04:30:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
04:30:01      dev8-48      1.10      1.71    279.46    256.00      0.00      0.37      0.36      0.04
04:30:01      dev8-32      1.11      2.13    282.45    256.00      0.00      0.36      0.36      0.04
04:30:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
04:30:01      dev8-64      1.09      0.43    277.76    256.00      0.00      0.34      0.34      0.04
04:30:01     dev253-0      0.21      0.00      3.15     14.99      0.00      0.00      0.00      0.00
04:30:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:40:01       dev8-0      0.22      0.00      3.41     15.17      0.00      0.00      0.00      0.00
04:40:01      dev8-16    200.43   4024.65   3663.24     38.36      0.04      0.18      0.17      3.45
04:40:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
04:40:01      dev8-48      1.11      2.13    281.16    256.00      0.00      0.33      0.33      0.04
04:40:01      dev8-32      1.11      2.13    282.44    256.00      0.00      0.33      0.33      0.04
04:40:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
04:40:01      dev8-64      1.10      1.71    279.03    256.00      0.00      0.33      0.33      0.04
04:40:01     dev253-0      0.24      0.00      3.41     14.22      0.00      0.00      0.00      0.00
04:40:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:40:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:50:01       dev8-0      0.19      0.00      2.92     15.64      0.00      0.00      0.00      0.00
04:50:01      dev8-16    198.58   4003.72   3672.62     38.66      0.04      0.18      0.17      3.44
04:50:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
04:50:01      dev8-48      1.11      2.56    281.59    256.00      0.00      0.37      0.37      0.04
04:50:01      dev8-32      1.11      2.56    282.44    256.00      0.00      0.34      0.34      0.04
04:50:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
04:50:01      dev8-64      1.09      0.85    278.60    256.00      0.00      0.34      0.34      0.04
04:50:01     dev253-0      0.19      0.00      2.92     15.10      0.00      0.00      0.00      0.00
04:50:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:50:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:00:01       dev8-0      0.21      0.00      3.11     15.07      0.00      0.00      0.00      0.00
05:00:01      dev8-16    201.29   4028.65   3690.12     38.35      0.04      0.18      0.17      3.46
05:00:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
05:00:01      dev8-48      1.10      1.71    279.46    256.00      0.00      0.34      0.34      0.04
05:00:01      dev8-32      1.11      2.13    282.87    256.00      0.00      0.34      0.33      0.04
05:00:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
05:00:01      dev8-64      1.09      0.85    277.32    256.00      0.00      0.31      0.31      0.03
05:00:01     dev253-0      0.21      0.00      3.11     14.49      0.00      0.00      0.00      0.00
05:00:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:00:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

this is the output of sar -d (although bear in mind that the cron is still running and so we’re not seeing the slow down currently or in this time range - would it help to run this when everything is grinding?)

05:00:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
05:10:01       dev8-0      0.23      0.00      3.48     14.94      0.00      0.00      0.00      0.00
05:10:01      dev8-16    202.43   3999.60   3671.59     37.89      0.04      0.18      0.17      3.51
05:10:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
05:10:01      dev8-48      1.10      1.28    279.44    256.00      0.00      0.35      0.35      0.04
05:10:01      dev8-32      1.12      3.41    282.43    256.00      0.00      0.38      0.38      0.04
05:10:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
05:10:01      dev8-64      1.09      1.28    278.59    256.00      0.00      0.34      0.34      0.04
05:10:01     dev253-0      0.25      0.00      3.48     14.13      0.00      0.00      0.00      0.00
05:10:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:10:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:20:01       dev8-0      0.19      0.00      3.00     15.79      0.00      0.00      0.00      0.00
05:20:01      dev8-16    209.87   4031.35   3749.00     37.07      0.04      0.17      0.17      3.48
05:20:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
05:20:01      dev8-48      1.10      1.71    280.31    256.00      0.00      0.35      0.35      0.04
05:20:01      dev8-32      1.10      0.43    281.16    256.00      0.00      0.36      0.36      0.04
05:20:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
05:20:01      dev8-64      1.09      1.28    278.60    256.00      0.00      0.32      0.32      0.04
05:20:01     dev253-0      0.19      0.00      3.00     15.38      0.00      0.00      0.00      0.00
05:20:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:20:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01       dev8-0      0.21      0.00      3.21     15.06      0.00      0.00      0.00      0.00
05:30:01      dev8-16    214.25   4168.75   3825.79     37.31      0.04      0.18      0.17      3.67
05:30:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
05:30:01      dev8-48      1.10      0.85    280.31    256.00      0.00      0.33      0.33      0.04
05:30:01      dev8-32      5.29     74.52    308.47     72.42      0.00      0.25      0.23      0.12
05:30:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.05      0.04      0.00
05:30:01      dev8-64      1.09      0.43    278.60    256.00      0.00      0.35      0.35      0.04
05:30:01     dev253-0      0.22      0.00      3.21     14.39      0.00      0.00      0.00      0.00
05:30:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:40:01       dev8-0      0.23      0.00      3.72     16.07      0.00      0.06      0.01      0.00
05:40:01      dev8-16    218.41   4181.00   3851.82     36.78      0.04      0.18      0.17      3.70
05:40:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
05:40:01      dev8-48      1.11      2.99    281.16    256.00      0.00      0.33      0.33      0.04
05:40:01      dev8-32     37.24    304.38    316.58     16.67      0.01      0.17      0.14      0.53
05:40:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
05:40:01      dev8-64      1.09      0.43    277.75    256.00      0.00      0.33      0.33      0.04
05:40:01     dev253-0      0.24      0.00      3.72     15.30      0.00      0.06      0.01      0.00
05:40:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:40:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:50:01       dev8-0      0.19      0.00      2.92     15.65      0.00      0.00      0.00      0.00
05:50:01      dev8-16    227.13   4245.76   3944.83     36.06      0.04      0.18      0.17      3.88
05:50:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
05:50:01      dev8-48      1.09      0.85    279.46    256.00      0.00      0.35      0.35      0.04
05:50:01      dev8-32     80.29    606.24    318.71     11.52      0.01      0.15      0.12      1.00
05:50:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
05:50:01      dev8-64      1.10      2.13    279.88    256.00      0.00      0.32      0.32      0.04
05:50:01     dev253-0      0.19      0.00      2.92     15.24      0.00      0.00      0.00      0.00
05:50:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:50:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

05:50:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
06:00:01       dev8-0      0.23      0.00      3.58     15.58      0.00      0.00      0.00      0.00
06:00:01      dev8-16    234.64   4297.08   4017.00     35.43      0.04      0.17      0.17      3.88
06:00:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
06:00:01      dev8-48      1.11      1.71    281.60    256.00      0.00      0.31      0.31      0.03
06:00:01      dev8-32    142.33   1007.24    317.01      9.30      0.02      0.15      0.13      1.79
06:00:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
06:00:01      dev8-64      1.09      0.43    278.61    256.00      0.00      0.34      0.33      0.04
06:00:01     dev253-0      0.24      0.00      3.58     14.83      0.00      0.00      0.00      0.00
06:00:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:00:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:10:01       dev8-0      0.23      0.00      3.45     15.22      0.00      0.00      0.00      0.00
06:10:01      dev8-16    234.76   4313.54   4035.93     35.57      0.04      0.19      0.18      4.15
06:10:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
06:10:01      dev8-48      1.12      3.41    282.86    256.00      0.00      0.36      0.36      0.04
06:10:01      dev8-32    126.76    905.57    319.12      9.66      0.02      0.16      0.14      1.73
06:10:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
06:10:01      dev8-64      1.10      2.13    279.02    256.00      0.00      0.34      0.34      0.04
06:10:01     dev253-0      0.25      0.00      3.45     13.89      0.00      0.00      0.00      0.00
06:10:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:10:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:20:01       dev8-0      0.18      0.00      2.89     15.90      0.00      0.00      0.00      0.00
06:20:01      dev8-16    258.81   4438.84   4159.86     33.22      0.04      0.17      0.16      4.19
06:20:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
06:20:01      dev8-48      1.12      3.41    283.30    256.00      0.00      0.32      0.32      0.04
06:20:01      dev8-32    272.12   1868.74    319.56      8.04      0.04      0.15      0.12      3.24
06:20:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.05      0.05      0.00
06:20:01      dev8-64      1.09      1.71    278.60    256.00      0.00      0.33      0.33      0.04
06:20:01     dev253-0      0.19      0.00      2.89     15.47      0.00      0.00      0.00      0.00
06:20:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:20:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:30:01       dev8-0      0.21      0.00      3.21     15.29      0.00      0.00      0.00      0.00
06:30:01      dev8-16    246.74   4523.67   4198.26     35.35      0.04      0.18      0.17      4.18
06:30:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.05      0.05      0.00
06:30:01      dev8-48      1.12      3.41    282.87    256.00      0.00      0.38      0.38      0.04
06:30:01      dev8-32    131.66    945.31    319.14      9.60      0.02      0.16      0.13      1.68
06:30:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
06:30:01      dev8-64      1.09      1.28    278.18    256.00      0.00      0.33      0.33      0.04
06:30:01     dev253-0      0.22      0.00      3.21     14.70      0.00      0.00      0.00      0.00
06:30:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:30:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:40:01       dev8-0      0.22      0.00      3.30     15.02      0.00      0.00      0.00      0.00
06:40:01      dev8-16    256.72   4593.23   4298.45     34.64      0.04      0.17      0.16      4.22
06:40:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
06:40:01      dev8-48      1.13      4.69    285.00    256.00      0.00      0.33      0.33      0.04
06:40:01      dev8-32    255.91   1710.99    318.70      7.93      0.04      0.15      0.12      3.18
06:40:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
06:40:01      dev8-64      1.09      0.85    278.60    256.00      0.00      0.30      0.30      0.03
06:40:01     dev253-0      0.23      0.00      3.30     14.16      0.00      0.00      0.00      0.00
06:40:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:40:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

06:40:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
06:50:01       dev8-0      0.14      0.00      2.55     17.77      0.00      0.00      0.00      0.00
06:50:01      dev8-16    266.51   4801.89   4408.60     34.56      0.05      0.18      0.18      4.69
06:50:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
06:50:01      dev8-48      1.11      2.56    282.87    256.00      0.00      0.34      0.34      0.04
06:50:01      dev8-32    489.09   3330.50    322.55      7.47      0.07      0.15      0.12      5.78
06:50:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
06:50:01      dev8-64      1.09      0.85    278.60    256.00      0.00      0.36      0.36      0.04
06:50:01     dev253-0      0.15      0.00      2.55     16.98      0.00      0.00      0.00      0.00
06:50:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:50:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:00:01       dev8-0      0.15      0.00      2.78     17.91      0.00      0.00      0.00      0.00
07:00:01      dev8-16    262.35   4962.85   4456.81     35.90      0.05      0.18      0.17      4.48
07:00:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
07:00:01      dev8-48      1.13      4.27    284.58    256.00      0.00      0.33      0.32      0.04
07:00:01      dev8-32    488.17   3351.12    315.30      7.51      0.07      0.15      0.12      5.88
07:00:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
07:00:01      dev8-64      1.09      1.28    277.75    256.00      0.00      0.34      0.34      0.04
07:00:01     dev253-0      0.16      0.00      2.78     17.18      0.00      0.00      0.00      0.00
07:00:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:00:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:10:01       dev8-0      0.18      0.00      3.05     16.62      0.00      0.00      0.00      0.00
07:10:01      dev8-16    256.92   4895.99   4376.38     36.09      0.05      0.18      0.17      4.47
07:10:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
07:10:01      dev8-48      1.12      2.99    283.28    256.00      0.00      0.36      0.36      0.04
07:10:01      dev8-32    377.24   2588.94    319.12      7.71      0.05      0.14      0.12      4.51
07:10:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.05      0.05      0.00
07:10:01      dev8-64      1.11      2.99    282.00    256.00      0.00      0.36      0.36      0.04
07:10:01     dev253-0      0.20      0.00      3.05     15.49      0.00      0.00      0.00      0.00
07:10:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:10:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:20:01       dev8-0      0.17      0.00      2.86     16.49      0.00      0.00      0.00      0.00
07:20:01      dev8-16    249.18   4896.32   4342.47     37.08      0.05      0.18      0.17      4.34
07:20:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
07:20:01      dev8-48      1.11      1.71    282.44    256.00      0.00      0.34      0.34      0.04
07:20:01      dev8-32    308.71   2095.75    317.00      7.82      0.05      0.15      0.13      3.87
07:20:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
07:20:01      dev8-64      1.09      0.85    278.18    256.00      0.00      0.32      0.32      0.03
07:20:01     dev253-0      0.18      0.00      2.86     16.18      0.00      0.00      0.00      0.00
07:20:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:20:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:30:01       dev8-0      0.21      0.00      3.22     15.11      0.00      0.00      0.00      0.00
07:30:01      dev8-16    240.50   4826.56   4271.22     37.83      0.04      0.18      0.18      4.24
07:30:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
07:30:01      dev8-48      1.10      1.28    280.31    256.00      0.00      0.35      0.35      0.04
07:30:01      dev8-32    293.98   1998.40    317.00      7.88      0.04      0.15      0.13      3.72
07:30:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
07:30:01      dev8-64      1.09      1.71    277.32    256.00      0.00      0.35      0.35      0.04
07:30:01     dev253-0      0.22      0.00      3.22     14.33      0.00      0.00      0.00      0.00
07:30:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:30:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

07:30:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
07:40:01       dev8-0      0.21      0.00      3.25     15.61      0.00      0.00      0.00      0.00
07:40:01      dev8-16    238.30   4790.12   4216.61     37.80      0.04      0.19      0.17      4.07
07:40:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
07:40:01      dev8-48      1.11      2.13    281.16    256.00      0.00      0.34      0.34      0.04
07:40:01      dev8-32    215.69   1511.15    319.99      8.49      0.03      0.14      0.12      2.60
07:40:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
07:40:01      dev8-64      1.09      1.28    279.03    256.00      0.00      0.32      0.32      0.03
07:40:01     dev253-0      0.22      0.00      3.25     15.01      0.00      0.00      0.00      0.00
07:40:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:40:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:50:01       dev8-0      0.20      0.00      3.07     15.10      0.00      0.00      0.00      0.00
07:50:01      dev8-16    238.64   4763.34   4199.54     37.56      0.04      0.18      0.18      4.21
07:50:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
07:50:01      dev8-48      1.11      2.56    282.02    256.00      0.00      0.33      0.33      0.04
07:50:01      dev8-32    285.71   1960.37    317.86      7.97      0.04      0.15      0.12      3.49
07:50:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
07:50:01      dev8-64      1.09      0.43    278.18    256.00      0.00      0.32      0.31      0.03
07:50:01     dev253-0      0.21      0.00      3.07     14.50      0.00      0.00      0.00      0.00
07:50:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:50:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:00:01       dev8-0      0.21      0.00      3.30     15.73      0.00      0.00      0.00      0.00
08:00:01      dev8-16    234.39   4742.95   4177.78     38.06      0.04      0.18      0.17      4.07
08:00:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
08:00:01      dev8-48      1.12      3.84    282.44    256.00      0.00      0.36      0.36      0.04
08:00:01      dev8-32    266.59   1819.73    312.74      8.00      0.04      0.15      0.12      3.27
08:00:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
08:00:01      dev8-64      1.09      1.28    278.60    256.00      0.00      0.32      0.32      0.04
08:00:01     dev253-0      0.22      0.00      3.30     14.90      0.00      0.00      0.00      0.00
08:00:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:00:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:10:01       dev8-0      0.23      0.00      3.41     14.64      0.00      0.00      0.00      0.00
08:10:01      dev8-16    230.97   4642.21   4106.75     37.88      0.04      0.18      0.18      4.05
08:10:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
08:10:01      dev8-48      1.09      0.85    279.44    256.00      0.00      0.33      0.33      0.04
08:10:01      dev8-32    181.56   1283.24    317.84      8.82      0.03      0.15      0.13      2.42
08:10:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
08:10:01      dev8-64      1.08      0.00    276.88    256.00      0.00      0.33      0.32      0.03
08:10:01     dev253-0      0.25      0.00      3.41     13.84      0.00      0.00      0.00      0.00
08:10:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:10:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:       dev8-0      0.20      0.00      3.36     16.43      0.00      0.00      0.00      0.00
Average:      dev8-16    213.61   4099.22   3739.91     36.70      0.04      0.18      0.17      3.69
Average:      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
Average:      dev8-48      1.10      2.02    280.26    256.00      0.00      0.34      0.34      0.04
Average:      dev8-32     81.51    559.77    293.78     10.47      0.01      0.15      0.13      1.02
Average:      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
Average:      dev8-64      1.09      1.20    278.28    256.00      0.00      0.33      0.33      0.04
Average:     dev253-0      0.22      0.00      3.36     15.56      0.00      0.01      0.00      0.00
Average:     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

This is the sar -d report for yesterday covering 14:00 - 15:00 - the cron was disabled just after 14:10 and the issues started at around 14:20 so you can see the changes hopefully:

14:00:01       dev8-0      0.23      0.00      3.43     15.01      0.00      0.00      0.00      0.00
14:00:01      dev8-16    581.23   7513.66   6200.11     23.59      0.09      0.16      0.15      8.53
14:00:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.02      0.02      0.00
14:00:01      dev8-48      1.12      4.69    282.87    256.00      0.00      0.36      0.35      0.04
14:00:01      dev8-32   1769.60  12400.20    318.71      7.19      0.27      0.15      0.10     17.33
14:00:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
14:00:01      dev8-64      1.09      2.13    277.32    256.00      0.00      0.31      0.31      0.03
14:00:01     dev253-0      0.24      0.00      3.43     14.49      0.00      0.00      0.00      0.00
14:00:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:00:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:10:02       dev8-0      0.27      0.00      3.92     14.71      0.00      0.00      0.00      0.00
14:10:02      dev8-16    579.64   7550.45   6235.32     23.78      0.09      0.16      0.15      8.58
14:10:02      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
14:10:02      dev8-48      1.11      2.56    282.01    256.00      0.00      0.33      0.33      0.04
14:10:02      dev8-32   1905.76  13450.40    322.54      7.23      0.30      0.16      0.10     18.24
14:10:02      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
14:10:02      dev8-64      1.10      2.56    279.45    256.00      0.00      0.32      0.31      0.03
14:10:02     dev253-0      0.28      0.00      3.92     14.01      0.00      0.00      0.00      0.00
14:10:02     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:10:02     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:20:01       dev8-0      0.19      0.00      2.94     15.17      0.00      0.00      0.00      0.00
14:20:01      dev8-16    568.68   7589.00   6153.20     24.17      0.09      0.16      0.14      8.13
14:20:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
14:20:01      dev8-48   1550.23  11514.64    272.66      7.60      0.24      0.16      0.09     14.69
14:20:01      dev8-32   4777.68  32148.58    292.74      6.79      0.75      0.16      0.08     40.24
14:20:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
14:20:01      dev8-64      1.06      2.56    268.39    256.00      0.00      0.34      0.34      0.04
14:20:01     dev253-0      0.20      0.00      2.94     14.79      0.00      0.00      0.00      0.00
14:20:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:20:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

14:20:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
14:30:01       dev8-0      0.23      0.00      3.41     15.07      0.00      0.00      0.00      0.00
14:30:01      dev8-16    413.49   7098.99   5562.69     30.62      0.07      0.17      0.16      6.46
14:30:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
14:30:01      dev8-48   5587.33  41171.82    286.71      7.42      0.87      0.16      0.09     47.79
14:30:01      dev8-32   6164.16  43928.35    299.51      7.18      0.96      0.16      0.08     52.15
14:30:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
14:30:01      dev8-64   1310.66  10425.17    286.71      8.17      0.20      0.15      0.10     13.09
14:30:01     dev253-0      0.23      0.00      3.41     14.53      0.00      0.00      0.00      0.00
14:30:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:30:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:40:01       dev8-0      0.25      0.00      3.68     14.84      0.00      0.07      0.01      0.00
14:40:01      dev8-16    573.52   6374.76   5632.33     20.94      0.09      0.15      0.14      7.90
14:40:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
14:40:01      dev8-48      1.10      1.71    280.74    256.00      0.00      0.30      0.30      0.03
14:40:01      dev8-32   1185.04   8339.13    323.83      7.31      0.18      0.15      0.10     11.78
14:40:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.04      0.04      0.00
14:40:01      dev8-64      1.10      2.99    278.61    256.00      0.00      0.32      0.32      0.04
14:40:01     dev253-0      0.26      0.00      3.68     14.08      0.00      0.07      0.01      0.00
14:40:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:40:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:50:01       dev8-0      0.25      0.00      4.36     17.20      0.00      0.00      0.00      0.00
14:50:01      dev8-16    622.65   7839.20   6414.72     22.89      0.10      0.15      0.14      8.78
14:50:01      dev8-80      0.42      3.47      0.00      8.32      0.00      0.03      0.03      0.00
14:50:01      dev8-48      1.11      2.13    281.16    256.00      0.00      0.32      0.32      0.04
14:50:01      dev8-32   1912.57  13774.94    327.24      7.37      0.30      0.16      0.10     18.44
14:50:01      dev8-96      0.42      3.47      0.00      8.32      0.00      0.05      0.05      0.00
14:50:01      dev8-64      1.08      0.43    275.19    256.00      0.00      0.31      0.31      0.03
14:50:01     dev253-0      0.28      0.00      4.36     15.57      0.00      0.00      0.00      0.00
14:50:01     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:50:01     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

This appears to be the peak disk utilization, and according to sar the access wait was adding about 0.16 ms to transactions. The disks have the same or higher await even in low throughput times.

These are recommended to be configured to the number of cores on the server. On newer releases, removing these configs will default to the number of cores.

I think you need to increase the number of transaction threads (since the problem arises when threads are blocked by higher io wait) . You could remove the configs and let the server pick based on your hardware or if you know this wouldn’t increase from 4 then configure transaction-queues x transaction-threads-per-queue to equal 64 (4x increase), such as 8,8 or 4,16.

Thanks @kporter - I’ll have a look at changing those values now and we can see if it makes any difference

Really appreciate the help on this.

If they aren’t in the service context then they should be hardware based already. So you will need to manually increase them. I think the threads per queue is dynamic now, if so, start there.

found them and changed - just restarting to check - had to manually set the config values as the transaction threads stayed at 4.

Sorry, I made a couple mistakes:

  1. I misread your original post and thought that the issue occurred when the cron job was running, but, as strange as it is, you are actually saying the issue occurs when the cron job is stopped.

    That is very stunning! Do your queries primarily read this data that the cron job is loading? If so, I believe you are benefitting from the cron job keeping the post write queue loaded with frequently queried data which acts as a cache. If so you would likely benefit from increasing the size of your post-write-queue.

  2. Adjusting the transaction-threads-per-queue will not improve query performance (unless you have enabled query-in-transaction-thread). Generally for queries you would need to increase query-worker-threads when they are bottlenecking on IO.

@Crags your actual aerospike.conf would be nice - I am particularly interested in the namespace/device configuration. As @kporter said its likely IO overload. When you load fresh data into aerospike, it will go into a post-write-queue which stores the newly written data in memory temporarily . If you have a process that refreshes this data, and another that goes and tries to pull it, if it still remains in the post-write-queue cache your IO strain will be reduced. So curious to understand 1) what does your device config look like 2) what hardware is it. If this is IO strain, querymicrobenchmarks should show that. You mentioned you did have microbenchmarks enabled and posted, but I don’t see any - microbenchmarks/querymicrobenchmarks are seperate.

No problem @kporter, but I think you’ve hit the nail exactly on the head. That would make sense if when we’re refreshing the data (currently every 5 minutes) the cache is speeding up performance. We figured their must have been some kind of cache system in effect but were unsure what or how, now we know.

I did try increasing the query-worker-threadsbut I’ll try that again when I next get chance and see if it improves performance. I think it reduced IO bottleneck slightly as a trade off for slightly slower queries.

Thanks again, very much appreciated.

Thanks @Albot, below is our current aerospike.conf so this will help to show you how we’re configured. We’re running 2 nodes, 12 core server with a decent amount of ram in each one (can’t remember exactly if they are R630’s or R730’s) - we have them maxed with 6 SSD’s although as our use case has changed we’re hardly using any disk space currently (like the temp01 namespace is using around 3GB and the adspruce one is using around 40Mb! )

I’m wondering if we would benefit from this being in-memory instead of persistent as then the IO issue would be removed and we have the data synced anyway from NySQL ?

# Aerospike database configuration file.

service {
	user root
	group root
	paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
	pidfile /var/run/aerospike/asd.pid
	#service-threads 4
	transaction-queues 8
	transaction-threads-per-queue 8
	proto-fd-max 70000
}

logging {
	# Log file must be an absolute path.
	file /var/log/aerospike/aerospike.log {
		context any info
	}
}

network {
        service {
                address any
                port 3000
                access-address 192.168.120.51
        }

        heartbeat {
                mode multicast
                multicast-group 239.1.99.51
                port 9918
                address 192.168.60.51

                interval 150
                timeout 10
        }

        fabric {
                address 192.168.60.51
                port 3001
        }

        info {
                address any
                port 3003
        }
}

namespace temp01 {
        replication-factor 2
        memory-size 32G
        default-ttl 86400		# 86400 = 24 hours

        storage-engine device {
                device /dev/sdb1
                scheduler-mode noop
                write-block-size 128K
        }
}

namespace adspruce {
        replication-factor 2
        memory-size 64G
        default-ttl 0			# 0 = no default TTL

        storage-engine device {
        device /dev/sdc1
	device /dev/sdd1
	device /dev/sde1
	scheduler-mode noop
        write-block-size 128K
        }
}

@Albot I enabled query-microbenchmarks not microbenchmarks and ran a histogram on it while it was running (which is here: Very strange Aerospike behaviour - completely unknown cause - #2 by Crags the post above)

Perhaps I’m running the wrong commands to get the data you need (I enabled it with asinfo -v "set-config:context=service;query-microbenchmark=true") so if that is the case, let me know what I can run to test this theory.

Thanks

Okay @rbotzer / @kporter / @Albot - in order to negate the possible IO issue, what I have done is converted the namespace to in memory only and restarted the nodes. There’s a definite improvement in queries and the IO issue seems to be resolved from the severs in the fact that I can no longer see it in CheckMK

  • however -

this has led to identifying another issue, in that it appears we are unable to conduct more than 1k queries per second per node - is this an aerospike limitation, or could it be simply that we’re trying to do too much with what we have? They are not taxing queries, basically running through around 400 records (each one is around 3k) querying a secondary index for a value of either 1 or 0 (numeric sindex) and then returning this data which can amount to around 700/800k in total - does that sounds like it’s doing too much?

Also, I have run benchmarks on the querying which shows as below:

{adspruce}-query
Sep 18 2017 14:21:26
               % > (ms)
slice-to (sec)      1      8     64    ops/sec
-------------- ------ ------ ------ ----------
14:21:36    10  11.60   0.81   0.02      936.5
14:21:46    10  10.96   0.60   0.13      950.3
14:21:56    10  10.58   0.31   0.00      924.5
14:22:06    10   9.36   0.44   0.00      939.6
14:22:16    10   8.64   0.49   0.00      929.6
14:22:26    10   8.42   0.69   0.00      937.0
14:22:36    10   7.76   0.69   0.34      935.3
14:22:46    10   7.61   0.33   0.01      955.6
14:22:56    10   7.27   0.45   0.00      965.6
14:23:06    10   6.72   0.55   0.00      937.7
14:23:16    10   7.71   0.60   0.00      970.6
14:23:26    10   6.38   0.31   0.00      946.4
14:23:36    10   4.89   0.10   0.00      936.8
14:23:46    10   4.23   0.09   0.00      932.9
14:23:56    10   4.26   0.11   0.00      967.9
14:24:06    10   3.64   0.17   0.00      955.0
14:24:17    11   4.04   0.24   0.00      871.7
14:24:27    10   3.26   0.10   0.00      933.5
14:24:37    10   3.47   0.12   0.00      960.8
14:24:47    10   3.69   0.21   0.00      952.9
14:24:57    10   3.71   0.09   0.00      951.1
14:25:07    10   3.57   0.46   0.00      968.1
14:25:17    10   4.23   0.55   0.00      986.2
14:25:27    10   4.23   0.58   0.00      984.3
14:25:37    10   4.12   0.36   0.00      977.7
14:25:47    10   4.15   0.28   0.00      947.8
14:25:57    10   4.09   0.57   0.00      952.7
14:26:07    10   3.89   0.28   0.00      966.3
14:26:17    10   3.77   0.33   0.00      939.9
14:26:27    10   3.84   0.35   0.00      975.4
14:26:37    10   4.26   0.74   0.00      978.5
14:26:47    10   4.04   0.34   0.00      941.7
14:26:57    10   3.10   0.24   0.00     1014.8
14:27:07    10   3.07   0.41   0.00      961.9
14:27:17    10  10.56   0.45   0.00      995.3
14:27:27    10  13.61   0.71   0.00      981.5
14:27:37    10  13.08   0.79   0.00      974.4
14:27:47    10  12.70   0.29   0.00      969.7
14:27:57    10  11.80   0.13   0.00      921.7
14:28:07    10   8.58   0.27   0.00      954.3
14:28:17    10  13.63   0.29   0.00      973.4
14:28:27    10  13.78   0.69   0.00     1001.7
14:28:37    10  13.17   0.40   0.00      982.2
14:28:47    10  14.31   0.45   0.01      913.4
14:28:57    10  12.64   0.29   0.00      969.7
14:29:07    10   8.89   0.69   0.02      997.7
14:29:17    10  13.68   0.72   0.01      957.7
14:29:27    10  13.25   0.41   0.00      973.1
14:29:37    10  15.43   0.51   0.00      954.0
14:29:47    10  13.75   0.62   0.02      960.2
14:29:57    10  15.99   1.17   0.00      944.7
14:30:07    10  12.83   0.91   0.00      982.2
14:30:17    10  14.39   0.44   0.00      941.4
14:30:27    10  15.64   0.44   0.00      959.7
14:30:37    10  15.75   0.48   0.00      940.8
14:30:47    10  14.52   0.18   0.00      940.9
14:30:57    10  14.57   0.36   0.00      919.4
14:31:07    10  12.53   0.36   0.01      979.8
14:31:17    10  14.21   0.37   0.01      955.1
14:31:27    10  15.87   0.50   0.01      967.7
14:31:37    10  14.55   0.30   0.00      936.8
14:31:47    10  12.97   0.11   0.00      971.8
14:31:57    10  12.55   0.19   0.00      947.9
14:32:07    10   9.34   0.24   0.00      972.4
14:32:17    10  14.16   0.13   0.00      970.8
14:32:27    10  14.16   0.14   0.00      965.7
14:32:37    10  12.20   0.18   0.01      956.7
14:32:47    10  14.27   0.39   0.00      968.5
14:32:57    10  13.52   0.41   0.00      932.7
14:33:07    10  11.30   0.25   0.01      962.2
14:33:17    10  11.29   0.58   0.00      996.4
14:33:27    10  13.02   0.31   0.00      978.1
14:33:37    10  13.98   0.73   0.00      947.9
14:33:47    10  15.23   0.22   0.01      949.4
14:33:57    10  13.15   0.17   0.00      908.3
14:34:07    10  10.11   0.38   0.00      958.3
14:34:17    10  16.22   0.47   0.00      938.8
14:34:27    10  15.90   0.74   0.00      937.7
14:34:37    10  15.45   0.67   0.00      926.7
14:34:47    10  12.78   0.16   0.00      965.1
14:34:57    10  12.76   0.25   0.00      937.4
14:35:07    10  11.47   0.40   0.00      941.5
14:35:17    10  13.75   0.32   0.00      917.6
14:35:27    10  14.10   0.28   0.00      953.6
14:35:37    10  11.20   0.33   0.00      950.5
14:35:47    10  12.99   0.29   0.00      923.3
14:35:57    10  12.88   0.46   0.00      949.8
14:36:07    10   9.26   0.42   0.00      935.1
14:36:17    10  15.45   0.23   0.00      931.9
14:36:27    10  16.92   0.16   0.00      948.6
14:36:37    10  13.50   0.33   0.00      918.3
14:36:47    10  13.91   0.29   0.00      924.6
14:36:57    10  13.53   0.89   0.00      951.1
14:37:07    10  13.10   0.74   0.00      927.3
14:37:17    10  12.70   0.50   0.00      983.7
14:37:27    10  14.20   0.33   0.00      945.9
14:37:37    10  13.48   0.40   0.00      941.8
14:37:47    10  13.84   0.23   0.00      916.5
14:37:57    10  12.50   0.16   0.00      931.7
14:38:07    10   9.63   0.21   0.01      923.2
14:38:17    10  17.22   0.14   0.00      939.2
14:38:27    10  14.57   0.37   0.00      922.6
14:38:37    10  12.72   0.92   0.01      944.0
14:38:47    10  14.52   0.19   0.00      911.1
14:38:57    10  13.59   0.15   0.00      893.4
14:39:07    10   9.40   0.35   0.00      949.2
14:39:17    10  14.63   0.24   0.00      906.9
14:39:27    10  17.58   0.43   0.00      887.0
14:39:37    10  15.16   0.37   0.00      910.5
14:39:47    10  12.75   0.42   0.00      921.7
14:39:57    10  12.80   0.59   0.00      928.8
14:40:07    10  11.24   0.67   0.00      928.3
14:40:17    10  15.64   0.48   0.00      936.5
14:40:27    10  14.77   0.24   0.00      940.8
14:40:37    10  14.31   0.31   0.00      921.1
14:40:47    10  13.55   0.61   0.00      956.7
14:40:57    10  14.66   0.63   0.00      943.6
14:41:07    10  10.84   0.16   0.00      924.1
14:41:17    10  16.15   0.55   0.00      931.4
-------------- ------ ------ ------ ----------
     avg        11.19   0.40   0.01      948.0
     max        17.58   1.17   0.34     1014.8

Are you being limited by the network? What you are describing would be approaching the limit of a gigabit network.

Run sar -n on both clients and servers to see their network utilization over time.