# Upgrade from 4.8.0.3 to 4.9.0.4, high rate of write timeout

**URL:** <https://discuss.aerospike.com/t/upgrade-from-4-8-0-3-to-4-9-0-4-high-rate-of-write-timeout/7357>\
**Category:** Upgrading\
**Created:** [April 20, 2020, 11:48am UTC](https://discuss.aerospike.com/t/upgrade-from-4-8-0-3-to-4-9-0-4-high-rate-of-write-timeout/7357 "2020-04-20T11:48:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![1114](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/1114/32/1411_2.png) [@1114](https://discuss.aerospike.com/u/1114)\
**Post date:** [April 20, 2020, 11:48am UTC](https://discuss.aerospike.com/t/upgrade-from-4-8-0-3-to-4-9-0-4-high-rate-of-write-timeout/7357/1 "2020-04-20T11:48:04Z")

</div>

For a rolling upgrade, I added a 4.9.0.4 (ip-10-1-14-204) node into the 4.8.0.3 cluster. Even after the partition migration done, the 4.9 node still had a very high rate of write timeout. Finally I removed it from the cluster and the performance of cluster recovered as usual.

 ![image](https://us1.discourse-cdn.com/flex019/uploads/aerospike/original/2X/d/daa941fa751ec96f8452e55283a540655da4f5d9.png)

All the nodes use the same configure.

I found logs like this on 4.8 nodes

```
Apr 20 2020 09:01:18 GMT: WARNING (rw): (replica_write.c:255) repl_write_handle_op: bad record
Apr 20 2020 09:01:18 GMT: WARNING (flat): (flat.c:183) unsupported storage fields
Apr 20 2020 09:01:18 GMT: WARNING (rw): (replica_write.c:255) repl_write_handle_op: bad record
Apr 20 2020 09:01:18 GMT: WARNING (flat): (flat.c:183) unsupported storage fields

```

this on 4.9 node

```
Apr 20 2020 09:04:36 GMT: WARNING (rw): (replica_write.c:418) repl-write ack: no digest 
Apr 20 2020 09:04:36 GMT: WARNING (rw): (replica_write.c:418) repl-write ack: no digest
Apr 20 2020 09:04:36 GMT: WARNING (rw): (replica_write.c:418) repl-write ack: no digest
Apr 20 2020 09:04:36 GMT: WARNING (rw): (replica_write.c:418) repl-write ack: no digest

```

I didn’t find any attention in the doc about upgrading from 4.8 to 4.9, but it seems not safe to do the rolling upgrade as usual. I succeeded with 4.6 to 4.7, 4.7 to 4.8 without any issue.

Any help?

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [April 20, 2020, 6:03pm UTC](https://discuss.aerospike.com/t/upgrade-from-4-8-0-3-to-4-9-0-4-high-rate-of-write-timeout/7357/2 "2020-04-20T18:03:04Z")

</div>

Could you share your `aerospike.conf` as well as the output of `asadm -e "info"`.

Are you running Aerospike Community or Enterprise?

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [April 20, 2020, 7:34pm UTC](https://discuss.aerospike.com/t/upgrade-from-4-8-0-3-to-4-9-0-4-high-rate-of-write-timeout/7357/3 "2020-04-20T19:34:49Z")

</div>

NVM we have confirmed that this is a bug in Aerospike Community Edition. We are working on a hot-fix.

BTW, this bug could have corrupted the data on that node, you should probably wipe that nodes disks and have it rejoin the 4.8 cluster. Migrations will repopulate the disks.

Thank you for reporting this issue.

---

<div class="post-metadata">

**Author:** ![1114](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/1114/32/1411_2.png) [@1114](https://discuss.aerospike.com/u/1114)\
**Post date:** [April 21, 2020, 1:24am UTC](https://discuss.aerospike.com/t/upgrade-from-4-8-0-3-to-4-9-0-4-high-rate-of-write-timeout/7357/4 "2020-04-21T01:24:01Z")

</div>

Yes, we are running Aerospike CE. Good to know you located the issue. Thank you for you remind, the 4.9 node has been terminated.

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/system/32/2274_2.png) [@system](https://discuss.aerospike.com/u/system)\
**Post date:** [April 27, 2020, 1:24am UTC](https://discuss.aerospike.com/t/upgrade-from-4-8-0-3-to-4-9-0-4-high-rate-of-write-timeout/7357/5 "2020-04-27T01:24:04Z")

</div>

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.
