# Aerospike Server: ASD process memory increases slowly with UDF list

**URL:** <https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528>\
**Category:** User Defined Functions (UDF)\
**Tags:** list, udf\
**Created:** [June 11, 2020, 12:08pm UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528 "2020-06-11T12:08:19Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![H\_Amp](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/h_amp/32/1452_2.png) [@H\_Amp](https://discuss.aerospike.com/u/H_Amp)\
**Post date:** [June 11, 2020, 12:08pm UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528/1 "2020-06-11T12:08:19Z")

</div>

We are seeing the memory of the “asd” process rising slowly over time in production and eventually we have to restart the Aerospike server.

We tried to isolate the issue and it looks like one of our UDFs is causing a leak. In order to reproduce the issue, we created a simplified version of our UDF and ran it in our test server.

**Server** : Community Edition - 4.8.0.5

**Test UDF**

```
local function createRecord(rec, requestId)
    rec["REQS_LIST"] = list {requestId}
    record.set_ttl(rec, 300)
    aerospike:create(rec)
end

local function updateRecord(rec)
    record.set_ttl(rec, 300)
    aerospike:update(rec)
end

function addRequest(rec, requestId)
    if (not aerospike:exists(rec)) then
        createRecord(rec, requestId)
    else
        local requestsList = rec["REQS_LIST"]
        list.append(requestsList, requestId)
        rec["REQS_LIST"] = requestsList
        updateRecord(rec)
    end
end

function removeRequest(rec)
    local requestsList = rec["REQS_LIST"]
    list.remove(requestsList, 1)
    rec["REQS_LIST"] = requestsList
    updateRecord(rec)
end

```

* * *

Our test script calls “addRequest”, waits for 0-500 ms randomly and calls “removeRequest” repeatedly for 1000 different keys (1000 Aerospike entries) in a loop, in total (add and then remove) 17million times, 220 per second. The memory increased from 39M to 1.1G in ~21 hours.

**Start and End log from “top” command:**

```
                           PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
Wed Jun 10 14:42:01 UTC 2020 - 15186 root 20 0 2266m 39m 4836 S 2.0 0.1 0:00.22 asd
Wed Jun 10 14:43:01 UTC 2020 - 15186 root 20 0 2293m 42m 5048 S 4.0 0.1 0:01.41 asd
Wed Jun 10 14:44:01 UTC 2020 - 15186 root 20 0 2301m 43m 5112 S 4.0 0.1 0:03.47 asd
...
Thu Jun 11 11:46:01 UTC 2020 - 15186 root 20 0 3363m 1.1g 5176 S 2.0 3.4 35:31.77 asd
Thu Jun 11 11:47:01 UTC 2020 - 15186 root 20 0 3363m 1.1g 5176 S 4.0 3.4 35:33.38 asd
Thu Jun 11 11:48:01 UTC 2020 - 15186 root 20 0 3363m 1.1g 5176 S 2.0 3.4 35:35.02 asd

```

Aerospike summary shows the memory used at 1% from start to end.

Please let me know if you need any info.

---

<div class="post-metadata">

**Author:** ![H\_Amp](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/h_amp/32/1452_2.png) [@H\_Amp](https://discuss.aerospike.com/u/H_Amp)\
**Post date:** [June 17, 2020, 2:30am UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528/2 "2020-06-17T02:30:19Z")

</div>

@pgupta Is it a known issue? Can you please take a look into this?

---

<div class="post-metadata">

**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\
**Post date:** [June 20, 2020, 2:33am UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528/3 "2020-06-20T02:33:01Z")

</div>

@H_Amp - We are not aware of any memory leak issues. Try using lua’s collectgarbage() to periodically check lua’s memory consumption. [Lua 5.3 Reference Manual](http://www.lua.org/manual/5.3/manual.html#pdf-collectgarbage)  
You can also use the function to perform immediate garbage collection.

---

<div class="post-metadata">

**Author:** ![H\_Amp](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/h_amp/32/1452_2.png) [@H\_Amp](https://discuss.aerospike.com/u/H_Amp)\
**Post date:** [June 22, 2020, 11:27am UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528/4 "2020-06-22T11:27:23Z")

</div>

@pgupta Thanks for the reply.

> [@pgupta](#):
>
> Try using lua’s collectgarbage() to periodically check lua’s memory consumption.

I added collectgarbage in the UDF but the “asd” process memory still kept on rising. The memory usage as shown by the collectgarbage(“count”) is always ~60 KB.

> [@pgupta](#):
>
> We are not aware of any memory leak issues.

I had done some testing on different scenarios before. I didn’t see leak when I had “list.append” and “list.remove” in the same function (addRequest). I only saw leak with my above UDF where I am adding in the list from one function and removing from the list from another function.

---

<div class="post-metadata">

**Author:** ![H\_Amp](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/h_amp/32/1452_2.png) [@H\_Amp](https://discuss.aerospike.com/u/H_Amp)\
**Post date:** [August 11, 2020, 12:56am UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528/5 "2020-08-11T00:56:47Z")

</div>

@pgupta Any updates on this? I have attached the sample script and “top” output above showing the memory rising constantly in my test server. Let me know if you need anything else to confirm the issue.

We have to restart the Aerospike servers every few weeks because of this.

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [August 11, 2020, 4:34pm UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528/6 "2020-08-11T16:34:53Z")

</div>

Looking at release notes there were some ordered list issues that were fixed in later versions. But nothing with a UDF leak. Please see release notes: [Aerospike Server CE Release Note | Download | Aerospike](https://www.aerospike.com/download/server/notes.html#4.8.0.5), does this occur on latest 4.8, or 4.9? Could you test on latest 5.1 with lots of UDF improvements. And please confirm if using secondary index?

---

<div class="post-metadata">

**Author:** ![H\_Amp](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/h_amp/32/1452_2.png) [@H\_Amp](https://discuss.aerospike.com/u/H_Amp)\
**Post date:** [August 12, 2020, 12:53am UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528/7 "2020-08-12T00:53:07Z")

</div>

@lucien I have not tested with 4.9 or 5.1 as the release notes did not mention about any memory leak fixes. I will test with both and let you know if I still see the leak.

---

<div class="post-metadata">

**Author:** ![H\_Amp](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/h_amp/32/1452_2.png) [@H\_Amp](https://discuss.aerospike.com/u/H_Amp)\
**Post date:** [August 20, 2020, 5:18am UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528/8 "2020-08-20T05:18:40Z")

</div>

@lucien 4.9 is not leaking my test server. We will upgrade the production servers and reopen the issue if we see errors there.

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/system/32/2274_2.png) [@system](https://discuss.aerospike.com/u/system)\
**Post date:** [August 26, 2020, 5:18am UTC](https://discuss.aerospike.com/t/aerospike-server-asd-process-memory-increases-slowly-with-udf-list/7528/9 "2020-08-26T05:18:53Z")

</div>

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.