# High memory usage of Aerospike Server while performing UDF operations via aerospike golang client

**URL:** <https://discuss.aerospike.com/t/high-memory-usage-of-aerospike-server-while-performing-udf-operations-via-aerospike-golang-client/8958>\
**Category:** General Discussion\
**Tags:** udf\
**Created:** [November 4, 2021, 10:21am UTC](https://discuss.aerospike.com/t/high-memory-usage-of-aerospike-server-while-performing-udf-operations-via-aerospike-golang-client/8958 "2021-11-04T10:21:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![shahaashit](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/shahaashit/32/1566_2.png) [@shahaashit](https://discuss.aerospike.com/u/shahaashit)\
**Post date:** [November 4, 2021, 10:21am UTC](https://discuss.aerospike.com/t/high-memory-usage-of-aerospike-server-while-performing-udf-operations-via-aerospike-golang-client/8958/1 "2021-11-04T10:21:09Z")

</div>

Observing high memory usage of Aerospike Server while performing UDF operations via aerospike golang client.

After every execution of the udf function from client, memory keeps on increasing and ultimately the aerospike crashes after memory has exhausted.

**Methods already tried**

1. set “cache-enabled” property of “mod-lua” in config to “false” - didn’t help.
2. tried using predicate filtering (deprecated in latest aerospike version) instead of filter expression - didn’t help.
3. Tried with an empty udf function as well to eliminate the possibility of memory leak via udf function - didn’t help.
4. If I just query data (using client method “Query”) instead of executing udf function to delete them, memory isn’t impacted.

**Steps to replicate -**

1. Use filter expression and filter for a bin value.
2. Execute function “ExecuteUDF” of aerospike go client with the above filter while passing udf function to execute - “invalidate” (code mentioned below).

**Aerospike Memory stats**  
**Note** : all the memory snapshots are taken while there is no external traffic to the aerospike instance except for the script given below which is used for testing this.

Command used - **asadm -e “show stat” | grep mem**

1. Initial details before executing any query from client

```auto
memory_data_bytes|0
cluster_is_member |true
system_free_mem_pct |98
high-water-memory-pct |90
memory-size |4294967296
memory_free_pct |99
memory_used_bytes |19226816
memory_used_data_bytes |0
memory_used_index_bytes |19226816
memory_used_sindex_bytes |0
storage-engine.data-in-memory |false
xmem_id |0

```

1. After running below code once (observation - system\_free\_mem\_pct has reduced)

```auto
memory_data_bytes|0
cluster_is_member |true
system_free_mem_pct |79
high-water-memory-pct |90
memory-size |4294967296
memory_free_pct |99
memory_used_bytes |19226752
memory_used_data_bytes |0
memory_used_index_bytes |19226752
memory_used_sindex_bytes |0
storage-engine.data-in-memory |false
xmem_id |0

```

1. After running below code once again (observation - system\_free\_mem\_pct has reduced further)

```auto
memory_data_bytes|0
cluster_is_member |true
system_free_mem_pct |60
high-water-memory-pct |90
memory-size |4294967296
memory_free_pct |99
memory_used_bytes |19226752
memory_used_data_bytes |0
memory_used_index_bytes |19226752
memory_used_sindex_bytes |0
storage-engine.data-in-memory |false
xmem_id |0

```

1. After restarting aerospike (its back to the initial state mentioned in #1)

```auto
memory_data_bytes|0
cluster_is_member |true
system_free_mem_pct |98
high-water-memory-pct |90
memory-size |4294967296
memory_free_pct |99
memory_used_bytes |19226816
memory_used_data_bytes |0
memory_used_index_bytes |19226816
memory_used_sindex_bytes |0
storage-engine.data-in-memory |false
xmem_id |0

```

**Client code**  
Objective of this code - delete record where “abc” = “pqr9999” (“abc” is bin name here)

```auto
package main

import (
	"fmt"
	aero "github.com/aerospike/aerospike-client-go"
	"time"
)

func getClient() (*aero.Client, error) {
	policy := aero.ClientPolicy{
		Timeout: 20 * time.Second,
		IdleTimeout: 50 * time.Second,
		ConnectionQueueSize: 100,
		MinConnectionsPerNode: 50,
	}

	return aero.NewClientWithPolicy(&policy, "<ip>", 3000)
}

func aeroDelete(client *aero.Client) error {
	ns := "test-ssd"
	set := "default_set"

	stm := aero.NewStatement(ns, set)
	queryPolicy := aero.NewQueryPolicy()

	queryPolicy.FilterExpression = aero.ExpEq(
		aero.ExpStringBin("abc"),
		aero.ExpStringVal("pqr9999"),
	)

	task, err := client.ExecuteUDF(queryPolicy, stm, "delrec", "invalidate")
	if err == nil {
		for err := range task.OnComplete() {
			if err != nil {
				return err
			}
		}
	} else {
		return err
	}
	return nil
}

func main() {
	client, err := getClient()
	if err == nil {
		err = aeroDelete(client)
	}

	if err != nil {
		fmt.Println("error: ", err)
	}

	fmt.Println("done")
}

```

**UDF function definition**

```auto
function invalidate(rec)
        aerospike:remove(rec)
end

```

**Aerospike instance details -**

- Aerospike server version - 5.5.0.3
- Aerospike client (golang) version - 4.5.2
- Single Node aerospike
- All bins are non-indexed
- Having single namespace - “test-ssd” which has a single set “default\_set” having “300419” objects ![image](https://us1.discourse-cdn.com/flex019/uploads/aerospike/original/2X/a/ae094e52c302f0e0baa4b53e718b6f1e1fb42fd3.png)

**Config file**

```auto
service {
  user root
  group root
  batch-max-buffers-per-queue 512
  migrate-max-num-incoming 5
  migrate-threads 1
  paxos-single-replica-limit 1
  proto-fd-idle-ms 70000
  proto-fd-max 100000
}

namespace test-ssd {
  memory-size 4G
  allow-ttl-without-nsup true
  default-ttl 30D
  high-water-disk-pct 80
  high-water-memory-pct 90
  nsup-period 120
  replication-factor 2
  stop-writes-pct 100
  storage-engine device {
    defrag-lwm-pct 50
    device /dev/nvme0n1p1 /dev/sdb
    max-write-cache 128M
    read-page-cache true
    write-block-size 1M
  }
  background-scan-max-rps 100000
}

```

Can someone help with this please?

---

<div class="post-metadata">

**Author:** ![meher](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/meher/32/1644_2.png) [@meher](https://discuss.aerospike.com/u/meher)\
**Post date:** [November 8, 2021, 9:36pm UTC](https://discuss.aerospike.com/t/high-memory-usage-of-aerospike-server-while-performing-udf-operations-via-aerospike-golang-client/8958/2 "2021-11-08T21:36:33Z")

</div>

Seems this is a known issue that has been addressed in April in [version 5.5.0.9](https://download.aerospike.com/download/server/notes.html#5.5.0.9):

- [AER-6418] - (UDF) For namespaces with data not in memory, executing a UDF with an expression filter that evaluates false will leak memory.

---

<div class="post-metadata">

**Author:** ![shahaashit](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/shahaashit/32/1566_2.png) [@shahaashit](https://discuss.aerospike.com/u/shahaashit)\
**Post date:** [November 9, 2021, 8:43am UTC](https://discuss.aerospike.com/t/high-memory-usage-of-aerospike-server-while-performing-udf-operations-via-aerospike-golang-client/8958/3 "2021-11-09T08:43:52Z")

</div>

Thanks a lot @meher . Upgrading aerospike server to v5.7.0.8 (current latest) helped.

---

<div class="post-metadata">

**Author:** ![meher](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/meher/32/1644_2.png) [@meher](https://discuss.aerospike.com/u/meher)\
**Post date:** [November 10, 2021, 1:12am UTC](https://discuss.aerospike.com/t/high-memory-usage-of-aerospike-server-while-performing-udf-operations-via-aerospike-golang-client/8958/4 "2021-11-10T01:12:58Z")

</div>

Thanks for closing the loop!

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/system/32/2274_2.png) [@system](https://discuss.aerospike.com/u/system)\
**Post date:** [February 2, 2022, 1:13am UTC](https://discuss.aerospike.com/t/high-memory-usage-of-aerospike-server-while-performing-udf-operations-via-aerospike-golang-client/8958/5 "2022-02-02T01:13:20Z")

</div>

This topic was automatically closed 84 days after the last reply. New replies are no longer allowed.