# Aerospike Increased write Latency with I3 series EC2 Boxes with nvme disks **URL:** \ **Category:** Operations\ **Created:** [March 8, 2017, 12:22pm UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917 "2017-03-08T12:22:14Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Arpit\_Wanchoo](https://avatars.discourse-cdn.com/v4/letter/a/bbce88/32.png) [@Arpit\_Wanchoo](https://discuss.aerospike.com/u/Arpit_Wanchoo)\ **Post date:** [March 8, 2017, 12:22pm UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/1 "2017-03-08T12:22:14Z")
We have a 8 node cluster running in DRAM+DISK mode (I2.xlarge) instances . Was giving good performance with 99% calls within 1ms Yesterday we replaced 1 node with I3.xlarge and cpu wait time was even less on the new box. Today we replaced two more boxes with I3.xlarge and since then we have seen a degradation in write performance **on the all nodes** (old and new). 50% writes have gone beyond 1ms upto 4-8 ms bucket ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [March 8, 2017, 6:45pm UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/2 "2017-03-08T18:45:51Z")
1. What version of Aerospike are you running? 2. Please share your configuration. ---
**Author:** ![Arpit\_Wanchoo](https://avatars.discourse-cdn.com/v4/letter/a/bbce88/32.png) [@Arpit\_Wanchoo](https://discuss.aerospike.com/u/Arpit_Wanchoo)\ **Post date:** [March 8, 2017, 9:08pm UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/3 "2017-03-08T21:08:51Z")
We are using a patch build provided over version 3.9.1 ( 3.9.1-158-g1e8db6e) Config : ```auto service { user root group root paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1. pidfile /var/run/aerospike/asd.pid ## SET TO NUMBER OF CORES ## service-threads 4 transaction-queues 4 scan-threads 4 ########################### ## DONT CHANGE ## transaction-threads-per-queue 3 proto-fd-idle-ms 600000 proto-fd-max 100000 batch-max-requests 10000 migrate-threads 2 replication-fire-and-forget true ########################## } logging { file /var/log/aerospike/aerospike.log { context any info } } network { service { address any port 3000 } heartbeat { mode mesh port 3002 mesh-seed-address-port 10.0.23.154 3002 mesh-seed-address-port 10.0.23.95 3002 mesh-seed-address-port 10.0.23.94 3002 mesh-seed-address-port 10.0.23.89 3002 mesh-seed-address-port 10.0.23.190 3002 mesh-seed-address-port 10.0.23.164 3002 mesh-seed-address-port 10.0.23.144 3002 mesh-seed-address-port 10.0.23.219 3002 interval 150 timeout 20 } fabric { port 3001 } info { port 3003 } } namespace userdata { replication-factor 2 #### CHANGE FOR INSTANCE ### memory-size 25G ############################ default-ttl 0 # 30 days, use 0 to never expire/evict. storage-engine device { ## COLD START AND NO SHADOW DEVICE ## cold-start-empty true device /dev/nvme0n1 ##################################### ### 1MB FOR INSTANCE STORE ### write-block-size 1024K ############################# max-write-cache 1024M } # storage-engine memory } namespace user_config_data { replication-factor 2 memory-size 1G default-ttl 0 storage-engine device { cold-start-empty true file /dev/aerospike/user_config_data.dat filesize 1G write-block-size 1024K } ``` ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [March 9, 2017, 6:27pm UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/4 "2017-03-09T18:27:50Z")
There seems to be a missing closing brace at the end of your config, is this a typo? ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [March 9, 2017, 6:35pm UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/5 "2017-03-09T18:35:00Z")
The [replication-fire-and-forget](http://www.aerospike.com/docs/reference/configuration#replication-fire-and-forget) configuration was deprecated in [3.9.0](http://www.aerospike.com/download/server/notes.html#3.9.0). Are all nodes (new and old) running 3.9.1? Assuming the older nodes were are using fire-and-forget, this could explain the increase in observed write latency. The closest configuration to fire-and-forget is [write-commit-level-override master](http://www.aerospike.com/docs/reference/configuration#write-commit-level-override) which would need to be added to your namespaces. ---
**Author:** ![Arpit\_Wanchoo](https://avatars.discourse-cdn.com/v4/letter/a/bbce88/32.png) [@Arpit\_Wanchoo](https://discuss.aerospike.com/u/Arpit_Wanchoo)\ **Post date:** [March 10, 2017, 3:44am UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/6 "2017-03-10T03:44:11Z")
All nodes are on 3.9.1 since last 4 months. I guess the reason for latency increase in other nodes could be slowness in replication on the new nodes but that will be only if a client write waits for replication as well which is not set in our client app ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [March 10, 2017, 3:54am UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/7 "2017-03-10T03:54:00Z")
The default behaviour is that the server waits for replication before acking the client. To clarify, are you saying that your clients specify `write-commit-level master` for each transaction? ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [March 10, 2017, 4:11am UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/8 "2017-03-10T04:11:24Z")
Also there are several microbenchmarks that can assist in pinning down the exact source of latency: See [http://www.aerospike.com/docs/operations/monitor/latency](http://www.aerospike.com/docs/operations/monitor/latency). ---
**Author:** ![Arpit\_Wanchoo](https://avatars.discourse-cdn.com/v4/letter/a/bbce88/32.png) [@Arpit\_Wanchoo](https://discuss.aerospike.com/u/Arpit_Wanchoo)\ **Post date:** [March 14, 2017, 7:23am UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/9 "2017-03-14T07:23:58Z")
The performance was restored to normal after replacing all I3.xlarge back with I2.xlarge nodes. So this performance degradation has to do something with I3 box or configuring I3 box. Do we need any kind of pre warming of I3 disks ? Or could it be that new disk need the latest linux kernel ? Current version is : [ec2-user@ip-10-0-23-94 backup]$ uname -a Linux ip-10-0-23-94 4.4.14-24.50.amzn1.x86\_64 #1 SMP Fri Jun 24 19:56:04 UTC 2016 x86\_64 x86\_64 x86\_64 GNU/Linux ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [March 14, 2017, 5:01pm UTC](https://discuss.aerospike.com/t/aerospike-increased-write-latency-with-i3-series-ec2-boxes-with-nvme-disks/3917/10 "2017-03-14T17:01:01Z")
We recently published benchmarks for these instance types: [http://www.aerospike.com/benchmarks/amazon-ec2-i3-performance-results/](http://www.aerospike.com/benchmarks/amazon-ec2-i3-performance-results/)