Calling C module in LUA code


#1

I found some references to an article that used to exist going over calling C modules in LUA for a UDF function, but it appears to not exist anymore. I have created a simple helloworld module and it works while calling from LUA script, but in Aerospike it will not.

Modules Written in C Modules written in C can also be used in Aerospike UDFs.

The basic steps involved are:

Write and compile C the C functions into a “.so” file (a C UDF module). Install the module on your Aerospike Server. Any Lua function can import this C module with a “require” statement, and the C functions will be available.

[root@localhost power]# cat hellofunc.lua
#!/usr/bin/lua
function power()
        require("power")
        result=square(1.414213598)
        --print(result)
        warn(result)
        return 0
end

It was able to be registered after some fumbling with aerospike .so paths, but cannot execute. aql> register module "hellofunc.lua" OK, 1 module added.

aql> EXECUTE hellofunc.power() ON test.test WHERE PK = 1
Error: (100) /opt/aerospike/usr/udf/lua/hellofunc.lua:4: attempt to call global 'square' (a nil value)

#2

This seems to have worked… something on the scope of square. I’ll need to do more playing, but it’d be really nice to find that guide Aerospike wrote.

#!/usr/bin/lua
require("power")
local result=square(1.414213598)
function zpower()
        return result
end

and…

#!/usr/bin/lua
require("power")
local result=square
function zpower()
        return result(213)
end

#3

did you try: power.square() instead - trust that is the syntax to call functions inside another module

https://www.tutorialspoint.com/lua/lua_modules.htm


#4

Yes @pgupta tried that as well…

[root@localhost power]# aql
Aerospike Query Client
Version 3.15.1.2
C Client Version 4.3.0
Copyright 2012-2017 Aerospike. All rights reserved.
aql> register module "hellofunc.lua"
OK, 1 module added.

aql> EXECUTE hellofunc.zpower() ON test.test WHERE PK = 1
Error: (100) /opt/aerospike/usr/udf/lua/hellofunc.lua:5: attempt to index global 'power' (a nil value)

aql> ^C
[root@localhost power]# cat hellofunc.lua
#!/usr/bin/lua
require("power")
--local result=square
function zpower()
        return power.square(213)
end

#5

What happened to the Aerospike article on writing UDF’s that reference C modules? I desperately want to see it :slight_smile:


#6

It used to live here: https://docs.aerospike.com/pages/viewpage.action?pageId=3807960#LuaUDF–DevelopingLuaModules-ModuleswritteninC


#7

The syntax that I see on the link I cited is:

mymathmodule = require("mymath")
mymathmodule.add(10,20)

So, perhaps like so: (?)

mypower = require("power")
... 
return mypower.square(213)

#8
[root@localhost power]# cat hellofunc.lua
#!/usr/bin/lua
local power=require("power")
function zpower()
        return power.square(213)
end
....
aql> register module "hellofunc.lua"
OK, 1 module added.

aql> EXECUTE hellofunc.zpower() ON test.test WHERE PK = 1
Error: (100) /opt/aerospike/usr/udf/lua/hellofunc.lua:4: attempt to index upvalue 'power' (a boolean value)

and…

[root@localhost power]# cat hellofunc.lua
#!/usr/bin/lua
function zpower()
        local power=require("power")
        return power.square(213)
end
[root@localhost power]# aql
Aerospike Query Client
Version 3.15.1.2
C Client Version 4.3.0
Copyright 2012-2017 Aerospike. All rights reserved.
aql> register module "hellofunc.lua"
OK, 1 module added.

aql> EXECUTE hellofunc.zpower() ON test.test WHERE PK = 1
Error: (100) /opt/aerospike/usr/udf/lua/hellofunc.lua:4: attempt to index local 'power' (a boolean value)

aql>

#9

since you are setup to test … try mypower instead of power itself for local name … ???


#10

I see some reference here: https://www.aerospike.com/docs/udf/developing_lua_modules.html and then they ask you to follow: https://www.lua.org/pil/26.1.html


#11
[root@localhost power]# cat hellofunc.lua
#!/usr/bin/lua
function zpower()
        local cpower=require("power")
        return cpower.square(213)
end
[root@localhost power]# aql
Aerospike Query Client
Version 3.15.1.2
C Client Version 4.3.0
Copyright 2012-2017 Aerospike. All rights reserved.
aql> register module "hellofunc.lua"
OK, 1 module added.

aql> EXECUTE hellofunc.zpower() ON test.test WHERE PK = 1
Error: (100) /opt/aerospike/usr/udf/lua/hellofunc.lua:4: attempt to index local 'cpower' (a boolean value)

aql>

#12

Also what really bothers me is that it works in luascript, but not Aerospike :expressionless:

[root@localhost power]# ./hellofunc.2.lua
45369
[root@localhost power]# vim hellofunc.2.lua^C
[root@localhost power]# cat hellofunc.2.lua
#!/usr/bin/lua
require("power")
--function zpower()
    print(square(213))
--end
[root@localhost power]# ./hellofunc.2.lua
45369
[root@localhost power]# cp hellofunc.2.lua hellofunc3.lua
[root@localhost power]# vim !$
vim hellofunc3.lua
[root@localhost power]#
[root@localhost power]#
[root@localhost power]# cat hellofunc3.lua
#!/usr/bin/lua
require("power")
function zpower()
    return(square(213))
end
[root@localhost power]# aql
Aerospike Query Client
Version 3.15.1.2
C Client Version 4.3.0
Copyright 2012-2017 Aerospike. All rights reserved.
aql> register module "hellofunc3.lua"
OK, 1 module added.

aql> EXECUTE hellofunc.zpower() ON test.test WHERE PK = 1
Error: (100) /opt/aerospike/usr/udf/lua/hellofunc.lua:4: attempt to index local 'cpower' (a boolean value)

aql> EXECUTE hellofunc3.zpower() ON test.test WHERE PK = 1
Error: (100) /opt/aerospike/usr/udf/lua/hellofunc3.lua:4: attempt to call global 'square' (a nil value)

aql>

#13

Seems this is all we have (pretty short), not sure how much more the original had: https://docs.aerospike.com/docs/udf/developing_lua_modules.html#modules-written-in-c


#14

Well that leaves a bit to be desired… This seems helpful: https://github.com/aerospike/udf-pubsub What I am trying to do is create a UDF which will work with protobuffer. There is a C version of google protobuf which I am able to compile as a shared object, and utilize via lua. If I can get these things working I may be able to open up a whole world of possibilities with my data, as most of it is serialized with protobuffer…


#15

I’m running into an issue with passing blob data to lua. In the C module if i use printf, the blob is properly sent to stdout but when i use lua_pushstring it doesn’t appear to be preserved all of the time. Can lua handle a blob data type? What kind of lua does aerospike allow in UDF, does luaJit work?


#16

I’ve got something very odd happening. I think I made a mistake in the C code… I am trying to serialize a simple string in protobuffer format, and then pass that into a working deserialization program to determine if the serialization/variables were able to be handled properly in lua.

Lua:

[root@localhost amessage_lua]# cat amessage.lua
#!/usr/bin/lua
require("amsg")
result=dobf("123")
io.write(result)
--result=dobf("123")
--io.write(result)

Shared object code:

[root@localhost amessage_lua]# cat amsg.c
#include <lua.h>
#include <lauxlib.h>
#include <lualib.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include "amessage.pb-c.h"
#define MAX_MSG_SIZE 1024


static void stackDump (lua_State *L) {
        int i;
        int top = lua_gettop(L);
        for (i = 1; i <= top; i++) {  /* repeat for each level */
        int t = lua_type(L, i);
        switch (t) {
                  case LUA_TSTRING:  /* strings */
                    printf("LUA_TSTRING INDEX %d `%s'",i, lua_tostring(L, i));
                    break;

                  case LUA_TBOOLEAN:  /* booleans */
                    printf("LUATBOOLEAN: %s", lua_toboolean(L, i) ? "true" : "false");
                    break;

                  case LUA_TNUMBER:  /* numbers */
                    printf("LUA_TNUMBER: %g", lua_tonumber(L, i));
                    break;

                  default:  /* other values */
                    printf("DEFAULT: %s", lua_typename(L, t));
                    break;

        }
        printf(" || ");  /* put a separator */
        }
        printf("\n");  /* end the listing */
}

static int dobf (lua_State *L){
        AMessage msg = AMESSAGE__INIT; // AMessage
        void *buf;                     // Buffer to store serialized data
        unsigned len;                  // Length of serialized data
        msg.a = lua_tonumber(L, -1);
        lua_remove(L, 1);
        len = amessage__get_packed_size(&msg);
        buf = malloc(len);
        amessage__pack(&msg,buf);
        lua_pushstring(L, buf);//, len);
        free(buf); // Free the allocated serialized dobfer
        stackDump(L);
        return 1;
}

int luaopen_amsg (lua_State *L){
        lua_register(L,"dobf",dobf);
        return 0;
}

[root@localhost amessage_lua]# ./amessage.lua
LUA_TSTRING INDEX 1 {▒&▒' ||
{▒&▒[root@localhost amessage_lua]# ./amessage.lua
LUA_TSTRING INDEX 1 {▒▒0' ||
{▒▒0[root@localhost amessage_lua]# ./amessage.lua
LUA_TSTRING INDEX 1 {lSb' ||

Removing the stackdump function so i can print to stdout and try to deserialize the data…(and recompiling of course)

gcc -Wall -shared -fPIC -o amsg.so -I/usr/include -llua $(pkg-config --cflags 'libprotobuf-c >= 1.0.0') $(pkg-config --libs 'libprotobuf-c >= 1.0.0') amsg.c amessage.pb-c.c -Wl,-E


[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
error unpacking incoming message
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123

Why would it work so infrequently, or at all if something was wrong? Here’s where I feel like I messed something up in the C code… If I call the function TWICE before printing, it ALWAYS works.

[root@localhost amessage_lua]# cat amessage.lua
#!/usr/bin/lua
require("amsg")
result=dobf("123")
result=dobf("123")
io.write(result)

[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123
[root@localhost amessage_lua]# ./amessage.lua | ./amessage_des
Received: a=123

Here’s the protobuf example I’m trying to follow https://github.com/protobuf-c/protobuf-c/wiki/Examples . If this is compiled in just C binaries it works fine, but when it’s put into these shared objects and called by lua maybe something different has to happen? I get the feeling I’m not treating the luabuffer correctly.


#17

Hi @Albot,

I think we have a resolution for you. :slight_smile:

The following link points to the missing link you referenced in an earlier post:

https://github.com/citrusleaf/aerospike-tools/tree/updateMay2018/asql/examples/lua_c

After running into the same problem you described and the same error message, I was able to find and update the missing example with more verbiage about what is going on between lua and C.

The “attempt to call global ‘square’ (a nil value)” is basically a result of the lua script not being able to find the handle (if you will) to the .so code.

The key can be found in the .c code with the invocation of the luaL_openlib() function instead of the register() function that was being used.

Stefan


#18

Hey @Albot,

just realized that the link is a private repo, I will put the example code on a public repo, but in the interim here is the .c code snipet:

#include <lua.h>
#include <lauxlib.h>
#include <lualib.h>

static int go(lua_State * L) {
  int rtrn = lua_tonumber(L, -1);   /* Get the single number arg */
  lua_pushnumber(L,rtrn*rtrn);      /* square it and push the return */

  return 1;
}

static const struct luaL_reg golib [] = {
  {"go", go},
  {NULL, NULL}
};

int luaopen_power(lua_State * L) {
  luaL_openlib(L, "go", golib, 0);
  return 1;
}

and the corresponding lua

function call_go(r, a_number)
    local power = require("power")
    local rtn = power.go(a_number);
    info(rtn)
    return run
end

Stefan


#19

I have a working example of using the protobuf module, which is a Lua shim wrapped around a shared object, at rbotzer/aerospike-lua-examples.


#20

wow thanks guys :slight_smile: I eventually did get it working with a c shared object using https://github.com/protobuf-c/protobuf-c . I think using this luarocks plugin might make things a lot easier. I’ll try this out