Memory Leak using UDF

1

I am facing memory leaks writing/updating using UDF, where as async write/update without UDF works fine. Details are as follows, please revert if you need more information

Write/Update TPS : 100K TPS Java Client version : 4.1.7 Aerospike version : 4.0.0.5 and 4.2.0.4

function insert_data(rec, BIN_NAME)
   if aerospike:exists(rec) then
        else
        rec[BIN_NAME] = 1
        aerospike:create(rec)
    end
    rec = nil
    BIN_NAME = nil
end
2

I am also experience same issue, in my aerospike cluster the memory keeps on increasing slowly and is killed by OOM. This is only happening when i use UDF, i can help with how to reproduce this issue @meher can you please look

3

Could this be garbage collection backup and not a memory leak?

Could you try backing off of UDF usage for a period of time and see how that affects memory usage?

4

Hi Kevin Porter,

Thanks for reply. Once we reached aerospike’s allocated memory limit and continued writing further, we could see on aerospike dashboard memory was stable, however machine’s memory was contiguously increasing. After that we stopped our application hitting aerospike and waited to check if memory coming back to expected usage, but it wasn’t.

@kporter

5

Hi, so i was experimenting few things, i found that whenever I uncomment line

rec[bin] = val

RES (memory) keeps increasing for the asd process

Steps to Reproduce:

udf file(sample UDF: pyloader.lua):

function insert_data(rec, bin, val)
    rec[bin] = val
    return 0
end

python code(app.py):

from __future__ import print_function
import aerospike
import time

def insertViaLua(keyId):
    key = ('ABCD', 'TEST_PY', keyId)
    client.apply(key, "pyloader", "insert_data", ["", 1])

config = {
  'hosts': [ ('172.0.0.1', 3000) ]
}

if __name__ == '__main__':
    try:
        client = aerospike.client(config).connect()
    except:
        import sys
        print("failed to connect to the cluster with", config['hosts'])
        sys.exit(1)
    client.udf_put('./pyloader.lua')
    keyId=1;
    while True:
        insertViaLua(keyId)
        keyId=keyId+1
client.close()

I am running Aerospike inside Docker:

docker run --env="NAMESPACE=sample" --env="REPL_FACTOR=1" --env="DEFAULT_TTL=30d" --env="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" --env="AEROSPIKE_VERSION=4.2.0.3" --volume="/opt/aerospike-test/data:/opt/aerospike/data:rw" --volume="/opt/aerospike-test/conf:/opt/aerospike/etc:rw" --volume="/opt/aerospike/data" -p 3003:3003 -p 3001:3001 -p 3000:3000 -p 3002:3002 --restart=no --detach=true -t aerospike/aerospike-server --config-file /opt/aerospike/etc/aerospike.conf

Namespace Conf:

namespace ABCD {
  data-in-index true
  default-ttl 0
  high-water-disk-pct 90
  high-water-memory-pct 90
  memory-size 524288000
  replication-factor 2
  single-bin true
  stop-writes-pct 95
  storage-engine device {
    data-in-memory true
    device /dev/nvme1n1
    write-block-size 128K
  }
}

let me know if you need any thing else @kporter

6

We found a memory leak in UDF executions for single-bin namespaces. This will be fixed in upcoming patches to 4.2.0 and 4.1.0.

@spmahale were your leaks observed on single-bin namespaces?

7

Yes, Perfect. We saw memory leaks on single bin namespaces. Looking forward to upcoming patches. Thanks Brad :slight_smile:

8

The single-bin memory leak is fixed as of server versions 4.1.0.4, 4.2.0.7, and all 4.3 versions.

1 Like