# "Error Code 8: Server memory error"

**URL:** <https://discuss.aerospike.com/t/error-code-8-server-memory-error/6861>\
**Category:** Operations\
**Created:** [November 28, 2019, 3:06am UTC](https://discuss.aerospike.com/t/error-code-8-server-memory-error/6861 "2019-11-28T03:06:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![datami\_devops](https://avatars.discourse-cdn.com/v4/letter/d/e9c0ed/32.png) [@datami\_devops](https://discuss.aerospike.com/u/datami_devops)\
**Post date:** [November 28, 2019, 3:06am UTC](https://discuss.aerospike.com/t/error-code-8-server-memory-error/6861/1 "2019-11-28T03:06:59Z")

</div>

Hi,

We are using aerospike

```auto
root@rgsp-datami-as-saopaulo-1a1{~}# asinfo -v version
Aerospike Community Edition build 3.16.0.6

```

We are noticing following error:

> {“das”:{“key”:{“namespace”:“das”,“setName”:“pkgOverallAcc”,“digest”:“iDcO4tObCyY+20fauXVwvS7O5yw=”,“userKey”:{“object”:“tpe-sig-campaign”,“type”:3}},“operations”:[{“type”:“ADD”,“binName”:“nsp\_t”,“value”:{“object”:93157,“type”:1}},{“type”:“ADD”,“binName”:“nsp\_d”,“value”:{“object”:62108,“type”:1}},{“type”:“ADD”,“binName”:“sig\_t”,“value”:{“object”:93157,“type”:1}},{“type”:“ADD”,“binName”:“sig\_d”,“value”:{“object”:62108,“type”:1}},{“type”:“READ”,“binName”:“t”,“value”:{“object”:null,“type”:0}},{“type”:“ADD”,“binName”:“nfg\_t”,“value”:{“object”:93157,“type”:1}},{“type”:“ADD”,“binName”:“nfg\_d”,“value”:{“object”:62108,“type”:1}},{“type”:“MAP\_MODIFY”,“binName”:“nfg\_dos”,“value”:{“object”:“AEmTqANhbmRyb2lkzgABa+UA”,“type”:4}},{“type”:“MAP\_MODIFY”,“binName”:“nsp\_dos”,“value”:{“object”:“AEmTqANhbmRyb2lkzgABa+UA”,“type”:4}},{“type”:“MAP\_MODIFY”,“binName”:“nsp\_app”,“value”:{“object”:“AEmTuANjb21fYmJ2YV9ueHRfcGVydS83XzVfMc4AAWvlAA==”,“type”:4}}],“resultCode”:8,“exception”:{“cause”:null,“stackTrace”:[{“methodName”:“parseResult”,“fileName”:“ReadCommand.java”,“lineNumber”:111,“className”:“com.aerospike.client.command.ReadCommand”,“nativeMethod”:false},{“methodName”:“execute”,“fileName”:“SyncCommand.java”,“lineNumber”:84,“className”:“com.aerospike.client.command.SyncCommand”,“nativeMethod”:false},{“methodName”:“operate”,“fileName”:“AerospikeClient.java”,“lineNumber”:1268,“className”:“com.aerospike.client.AerospikeClient”,“nativeMethod”:false},{“methodName”:“operate”,“fileName”:“AerospikeRepository.java”,“lineNumber”:120,“className”:“com.datami.repository.AerospikeRepository”,“nativeMethod”:false},{“methodName”:“updateUsage”,“fileName”:“PackageOverallAccountingDAOAerospike.java”,“lineNumber”:40,“className”:“com.datami.dao.zmi.accounting.PackageOverallAccountingDAOAerospike”,“nativeMethod”:false},{“methodName”:“updateAccounting”,“fileName”:“PackageOverallAccountingFacade.java”,“lineNumber”:30,“className”:“com.datami.zmi.accounting.facade.PackageOverallAccountingFacade”,“nativeMethod”:false},{“methodName”:“executeBolt”,“fileName”:“PackageOverallAccountingBolt.java”,“lineNumber”:54,“className”:“com.datami.zmi.accounting.topology.PackageOverallAccountingBolt”,“nativeMethod”:false},{“methodName”:“execute”,“fileName”:“CustomBaseRichBolt.java”,“lineNumber”:47,“className”:“com.datami.storm.topology.CustomBaseRichBolt”,“nativeMethod”:false},{“methodName”:“invoke”,“fileName”:“executor.clj”,“lineNumber”:729,“className”:“org.apache.storm.daemon.executor$fn\_\_5030$tuple\_action\_fn\_\_5032”,“nativeMethod”:false},{“methodName”:“invoke”,“fileName”:“executor.clj”,“lineNumber”:461,“className”:“org.apache.storm.daemon.executor$mk\_task\_receiver$fn\_\_4951”,“nativeMethod”:false},{“methodName”:“onEvent”,“fileName”:“disruptor.clj”,“lineNumber”:40,“className”:“org.apache.storm.disruptor$clojure\_handler$reify\_\_4465”,“nativeMethod”:false},{“methodName”:“consumeBatchToCursor”,“fileName”:“DisruptorQueue.java”,“lineNumber”:482,“className”:“org.apache.storm.utils.DisruptorQueue”,“nativeMethod”:false},{“methodName”:“consumeBatchWhenAvailable”,“fileName”:“DisruptorQueue.java”,“lineNumber”:460,“className”:“org.apache.storm.utils.DisruptorQueue”,“nativeMethod”:false},{“methodName”:“invoke”,“fileName”:“disruptor.clj”,“lineNumber”:73,“className”:“org.apache.storm.disruptor$consume\_batch\_when\_available”,“nativeMethod”:false},{“methodName”:“invoke”,“fileName”:“executor.clj”,“lineNumber”:848,“className”:“org.apache.storm.daemon.executor$fn\_\_5030$fn\_\_5043$fn\_\_5096”,“nativeMethod”:false},{“methodName”:“invoke”,“fileName”:“util.clj”,“lineNumber”:484,“className”:“org.apache.storm.util$async\_loop$fn\_\_557”,“nativeMethod”:false},{“methodName”:“run”,“fileName”:“AFn.java”,“lineNumber”:22,“className”:“clojure.lang.AFn”,“nativeMethod”:false},{“methodName”:“run”,“fileName”:“Thread.java”,“lineNumber”:745,“className”:“java.lang.Thread”,“nativeMethod”:false}],“resultCode”:8,“inDoubt”:false,“message”:“Error Code 8: Server memory error”,“localizedMessage”:“Error Code 8: Server memory error”,“suppressed”:},“level”:“ERROR”,“timestamp”:“2019-11-27T08:37:30Z”,“component”:“das”,“msgType”:“Aerospike exception”,“suppressed”:}}

This is because low\_device\_percent value has gone down.

I have increased fragmentation from 50 to 55.

```auto
root@rgsp-datami-as-saopaulo-1a1{~}# grep defrag /etc/aerospike/aerospike.conf
    defrag-lwm-pct 55

```

How can i fix this. We do have 3 nodes. one node recovered after instance restart. remaining 2 nodes still throwing this issue.

Thanks, Datami DevOps Team.

---

<div class="post-metadata">

**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\
**Post date:** [November 28, 2019, 7:51am UTC](https://discuss.aerospike.com/t/error-code-8-server-memory-error/6861/2 "2019-11-28T07:51:31Z")

</div>

What troubleshooting have you done? Why are you on 3.16? What does iostat look like? Whats your defrag-sleep at?

---

<div class="post-metadata">

**Author:** ![datami\_devops](https://avatars.discourse-cdn.com/v4/letter/d/e9c0ed/32.png) [@datami\_devops](https://discuss.aerospike.com/u/datami_devops)\
**Post date:** [November 28, 2019, 7:55am UTC](https://discuss.aerospike.com/t/error-code-8-server-memory-error/6861/3 "2019-11-28T07:55:26Z")

</div>

Hi Albot,

As mentioned we are using community edition and we didnt upgrade to 4.x. All three nodes started stating writes disrupted as low\_device\_percent reached less than 5%. now i have configured:

evict-tenths-pct 10 defrag-lwm-pct 55 defrag-startup-minimum 5

in aerospike.conf and restarting service one instance at a time. now all three instances are in sync.

How can we increase the low\_device\_percent value.

---

<div class="post-metadata">

**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\
**Post date:** [November 28, 2019, 8:47am UTC](https://discuss.aerospike.com/t/error-code-8-server-memory-error/6861/4 "2019-11-28T08:47:51Z")

</div>

What’s your defrag sleep set to? Can you capture screenshot of `iostat -zmxty 60` on an affected system? Can you also grab last few lines of defrag-q per device from Aerospike log?

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [December 4, 2019, 6:55pm UTC](https://discuss.aerospike.com/t/error-code-8-server-memory-error/6861/5 "2019-12-04T18:55:47Z")

</div>

Could you run:

```auto
asadm -e "info"
asadm -e "summary"

```