Commonly used library of functions

Feature Discussion User Defined Functions (UDF)
1

We have several Lua scripts for different aggregations.

Each aggregation is in a separate file.

But for all files common there is the filter function, right now we have a redundant copy of that function in every script file.

Is there a way to i.e. include a commonly used library of functions?

2

You can require your core module from the other ones. Assuming the module name is ā€˜coremod.luaā€™:

local Core = require('coremod')

local x = Core.myfunc('something')

This is mentioned in the Developing UDF Modules section.

1 Like
3

Awesome, thanks a lot for your quick response Ronen :grinning:

4

Ok, have been too fast to announce success here.

The solution works fine in aql, but does not work with Java or C# client.

This is how the script looks like now:

function execute(stream, paramMap)
    local modUtility = require("modUtility")
    local modFilter = require("modFilter")
    local kpiMethod = require(modUtility.kpiMethod(paramMap["kpiMethod"]))

    local function create_map_result(rec)
        local result = modUtility.prepareResult(rec, paramMap)
        local kpiMap = rec["kpiMap"]
        return kpiMethod.calc(kpiMap, paramMap, result)
    end

    local function filter_stream(rec)
        return modFilter.commonFilter(rec, paramMap)
    end

    return stream : filter(filter_stream) : map(create_map_result)
end

As mentioned works perfectly in aql, but when used from Java client I do get following error message:

com.aerospike.client.AerospikeException: org.luaj.vm2.LuaError: aerospike:195 calculateKpi:2 module 'modUtility' not found: modUtility
	no field package.preload['modUtility']
	modUtility.lua
	no class 'modUtility'

All required modules are available in the same udf folder as calculateKpi.lua as well.

Isnā€™t requiring other scripts working in the Aerospike clients or did I miss a step?

5

Two things, can you post the output of show modules from AQL, and give a bit of context - a few lines of Java to show how youā€™re calling whichever function in that module.

Also, Iā€™m not sure yet, but execute may be a reserved name. Can you rename it slightly?

6

Iā€™ve prepared an example for you to reproduce the issue and shared it with you via Dropbox (Iā€™m not allowed to publicly share the data).

The shared zip contains everything you need to reproduce the issue:

  • a dump of a small set with data
  • all UDFs needed
  • a working aggregate command for aql
  • a Java test class for the same aggregation that leads to the above mentioned exception

Let me know if you need anything else.

TIA

BTW: renaming method name ā€œexecuteā€ to something else did not help

7

Update: Got this working now in C# client

Also figured out the problem seems to lie in LuaJ as described here:

The suggested solution to prefix the folder udf helps to find the required modules, but then ends up in another error message:

com.aerospike.client.AerospikeException: org.luaj.vm2.LuaError: aerospike:195 calculateKpi:2 load udf\modUtility.lua: org.luaj.vm2.LuaError: No undumper.

Not sure what that means and googling for LuaJ ā€œno undumperā€ did not really bring any helpful results, also prefixing the module names will break them in Aerospike and aql I guess, so generally this is probably not a good solution at all.

Would be great if you would have another solution.

Another update: Adding udf folder to the classpath also helps solving the ā€œmodule not foundā€ error, but still got stuck at the same ā€œno undumperā€ error message.

8

This is now resolved also in Java.

The problem actually occurred in the C# client as well, just not initially, but only when trying to execute a function within a ā€œrequiredā€ module in an added reduce step.

To solve this problem we needed to move the required modules into the corresponding functions, so the script above now looks this way:

function execute(stream, paramMap)

    local function create_map_result(rec)
        local modUtility = require("modUtility")
        local kpiMethod = require(modUtility.kpiMethod(paramMap["kpiMethod"]))
        local result = modUtility.prepareResult(rec, paramMap)
        local kpiMap = rec["kpiMap"]
        return kpiMethod.calc(kpiMap, paramMap, result)
    end

    local function filter_stream(rec)
        local modFilter = require("modFilter")
        return modFilter.commonFilter(rec, paramMap)
    end

    return stream : filter(filter_stream) : map(create_map_result)
end
1 Like