Commonly used library of functions

lua

#1

We have several Lua scripts for different aggregations.

Each aggregation is in a separate file.

But for all files common there is the filter function, right now we have a redundant copy of that function in every script file.

Is there a way to i.e. include a commonly used library of functions?


#2

You can require your core module from the other ones. Assuming the module name is ‘coremod.lua’:

local Core = require('coremod')

local x = Core.myfunc('something')

This is mentioned in the Developing UDF Modules section.


#3

Awesome, thanks a lot for your quick response Ronen :grinning:


#4

Ok, have been too fast to announce success here.

The solution works fine in aql, but does not work with Java or C# client.

This is how the script looks like now:

function execute(stream, paramMap)
    local modUtility = require("modUtility")
    local modFilter = require("modFilter")
    local kpiMethod = require(modUtility.kpiMethod(paramMap["kpiMethod"]))

    local function create_map_result(rec)
        local result = modUtility.prepareResult(rec, paramMap)
        local kpiMap = rec["kpiMap"]
        return kpiMethod.calc(kpiMap, paramMap, result)
    end

    local function filter_stream(rec)
        return modFilter.commonFilter(rec, paramMap)
    end

    return stream : filter(filter_stream) : map(create_map_result)
end

As mentioned works perfectly in aql, but when used from Java client I do get following error message:

com.aerospike.client.AerospikeException: org.luaj.vm2.LuaError: aerospike:195 calculateKpi:2 module 'modUtility' not found: modUtility
	no field package.preload['modUtility']
	modUtility.lua
	no class 'modUtility'

All required modules are available in the same udf folder as calculateKpi.lua as well.

Isn’t requiring other scripts working in the Aerospike clients or did I miss a step?


#5

Two things, can you post the output of show modules from AQL, and give a bit of context - a few lines of Java to show how you’re calling whichever function in that module.

Also, I’m not sure yet, but execute may be a reserved name. Can you rename it slightly?


#6

I’ve prepared an example for you to reproduce the issue and shared it with you via Dropbox (I’m not allowed to publicly share the data).

The shared zip contains everything you need to reproduce the issue:

  • a dump of a small set with data
  • all UDFs needed
  • a working aggregate command for aql
  • a Java test class for the same aggregation that leads to the above mentioned exception

Let me know if you need anything else.

TIA

BTW: renaming method name “execute” to something else did not help


Map.merge doesn't work with nested maps [Resolved]
#7

Update: Got this working now in C# client

Also figured out the problem seems to lie in LuaJ as described here: http://stackoverflow.com/questions/16845805/updated-luaj-unable-to-call-require-function-in-lua-script

The suggested solution to prefix the folder udf helps to find the required modules, but then ends up in another error message:

com.aerospike.client.AerospikeException: org.luaj.vm2.LuaError: aerospike:195 calculateKpi:2 load udf\modUtility.lua: org.luaj.vm2.LuaError: No undumper.

Not sure what that means and googling for LuaJ “no undumper” did not really bring any helpful results, also prefixing the module names will break them in Aerospike and aql I guess, so generally this is probably not a good solution at all.

Would be great if you would have another solution.

Another update: Adding udf folder to the classpath also helps solving the “module not found” error, but still got stuck at the same “no undumper” error message.


#8

This is now resolved also in Java.

The problem actually occurred in the C# client as well, just not initially, but only when trying to execute a function within a “required” module in an added reduce step.

To solve this problem we needed to move the required modules into the corresponding functions, so the script above now looks this way:

function execute(stream, paramMap)

    local function create_map_result(rec)
        local modUtility = require("modUtility")
        local kpiMethod = require(modUtility.kpiMethod(paramMap["kpiMethod"]))
        local result = modUtility.prepareResult(rec, paramMap)
        local kpiMap = rec["kpiMap"]
        return kpiMethod.calc(kpiMap, paramMap, result)
    end

    local function filter_stream(rec)
        local modFilter = require("modFilter")
        return modFilter.commonFilter(rec, paramMap)
    end

    return stream : filter(filter_stream) : map(create_map_result)
end