LUA aggregation with byte array (AER-3589)

Feature Discussion Aggregation
1

I would like to make a request in Aerospike similar than the following SQL query :

SELECT ts,inst,blob from db.table where inst = 'VALUE' and ts > 12234 and ts < 12245

bins are inserted according to this structure (in python):

{ ts : integer for a timestamp
inst : string
blob : bytearray }

I have a secondary index on “ts”. “blob” is a bytearray which is the result of a “zlib.compress” of a “json.dumps”.

As I understood, I need to create a LUA aggregation function. So here is the LUA code I wrote :

local function map_profile(record)
return map {ts=record.ts,inst=record.inst,blob=record.blob }
end

function check_instrument(stream,instrument)
local function filter_instrument(record)
return record.inst == instrument
end
return stream : filter(filter_instrument) : map(map_profile)
end

Then, I make the following AQL query :

aggregate profile.check_instrument('VALUE') on mynamespace.myset where ts between 12234 and 12245

which give me this response :

+-------------------------------------------+
| check_instr... |
+-------------------------------------------+
| {"inst":"VALUE", "ts":12235} |
| {"inst":"VALUE", "ts":12236} |
+-------------------------------------------+

where is my bytearray blob ? If a put a string in my blob instead of a bytearray, the query above give me this response :

+-------------------------------------------+
| check_instr... |
+----------------------------------------------------------+
| {"inst":"VALUE", "ts":12235, "blob" : "hello"} |
| {"inst":"VALUE", "ts":12236, "blob" : "hello"} |
+----------------------------------------------------------+

with the blob !

What I did not understand ? Could we get a bytearray with an aggregation in LUA ? How can I have my bytearray blob in response of an aggregation query ?

Thank you

aerospike-server-community 3.5.9-1 ubuntu

2

I am able to get the blob back. Not sure what is going wrong in your case.

aql> select * from test.demo
+------+------------+----------------+
| ts   | inst       | blob           |
+------+------------+----------------+
| 1234 | "VALUE"    | 68 65 6C 6C 6F |
| 1236 | "NONVALUE" | 68 65 6C 6C 6F |
| 1236 | "VALUE"    | 68 65 6C 6C 6F |
| 1235 | "VALUE"    | 68 65 6C 6C 6F |
| 1235 | "NONVALUE" | 68 65 6C 6C 6F |
| 1234 | "NONVALUE" | 68 65 6C 6C 6F |
+------+------------+----------------+
6 rows in set (0.068 secs)

aql> aggregate simpleagg.check_instrument('VALUE') on test.demo where ts between 1233 and 1235
+----------------------------------------------------------+
| check_instr...                                           |
+----------------------------------------------------------+
| Map("ts"->1234, "inst"->"VALUE", "blob"->68 65 6C 6C 6F) |
| Map("ts"->1235, "inst"->"VALUE", "blob"->68 65 6C 6C 6F) |
+----------------------------------------------------------+
2 rows in set (0.006 secs)
3

Thank you.

What is your Aerospike’s version ? I see that in your case the values are returned in a “Map”. I have a dictionary. I will have a look on how I can get a Map. In the mean time, could you please post your LUA code ?

Best regards.

4

I am using the 3.5.9 enterprise edition. But this should not make any difference. Actually the result that you are getting is also a map. i am using an older version of aql which is displaying it differently. I am using the same lua file that you are using. So, I am wondering what is the difference.

Can you share the output of ‘select * from test.demo where ts between 1233 and 1235’

5

Hi,

I have found my problem. This is because I use the python client which does not seem to handle the bytearray properly.

In python, when I insert “hello” in a bytearray, I get the following :

| "NONVALUE" | 1236 | 63 5F 5F 62 75 69 6C 74 69 6E 5F 5F 0A 62 79 74 65 61 72 72 61 79 0A 70 31 0A 28 56 68 65 6C 6C 6F 0A 53 27 6C 61 74 69 6E 2D 31 27 0A 74 52 70 32 0A 2E |

This decoded blob content is :

 "c__builtin__\nbytearray\np1\n(Vhello\nS'latin-1'\ntRp2\n."

All work fine like you when I insert using the C client. There is something strange or something I do not understand with the python client.

aql> aggregate simpleagg.check_instrument('VALUE') on test.demo where ts between 1233 and 1236 ;


+----------------------------------------------------+
| check_instr...                                     |
+----------------------------------------------------+
| {"inst":"VALUE", "ts":1234}                        |
| {"inst":"VALUE", "ts":1236}                        |
| {"inst":"VALUE", "ts":1235}                        |
| {"inst":"VALUE", "blob":68 65 6C 6C 6F, "ts":1236} |
+----------------------------------------------------+

Thanks for your help.

Best regards

6

Ahh, that explains things…I inserted data using node.js client. But can you share your python code snippet to insert blob. This could be a python client issue.

7

Yes, it would be nice to get your snippet of Python, but I already see what’s going on, and have a JIRA ticket associated with it.

1 Like
8

Hi,

Here is my python code

import aerospike
config = { 'hosts': [ ("192.168.11.12", 3000) ] }
client = aerospike.client(config).connect()
for ts,val in (( 1234 , "VALUE"),
               ( 1234 , "NONVALUE" ),
               ( 1235 , "VALUE"),   
               ( 1235 , "NONVALUE" ),
               ( 1236 , "NONVALUE"),
               ( 1236 , "VALUE") ) :

    key = ('test',  'demo' , "{}{}".format(ts,
                                           val))
    client.put(key,  { "ts" : ts, 
                       "inst" : val  , 
                       "blob" :  bytearray('hello')   })

client.close()

Thanks for your help.

Best regards

9

Thanks! We’re in progress on this.

10

Hi @del,

The JIRA number in which we are tracking your issue is AER-3589.

11

This is now part of the new 1.0.44 release of the Python client. Please verify that it works for you.

1 Like
12

Thank you, now it works fine.

Best Regards.