sum_example object not set to an instance of an object


#1

Hi… another question, if you would be so kind. I’m attempting to run the example: http://www.aerospike.com/docs/client/csharp/usage/query/aggregate.html . I’ve verified that the UDF is installed on the server from the aql, created the records and the secondary index and now, with the following code, I am attempting to run the query / aggregation. This is resulting in “Object not set to an instance of an object” and crashing out. It crashes at ResultSet…

Am I missing something?

public void Aerospike() { AerospikeClient client = new AerospikeClient(“192.168.2.11”, 3000); string ns = “test”; string setName = “demoset”; string indexName = “aggindex”; string binName = “aggbin”;

        Statement stmt = new Statement();
        stmt.SetNamespace(ns);
        stmt.SetSetName(setName);
        stmt.SetBinNames(binName);
        stmt.SetFilters(Filter.Range(binName, 4, 7));

        ResultSet rs = client.QueryAggregate(null, stmt, "sum_example", "sum_single_bin", Value.Get(binName));

        if (rs.Next())
        {
            double sum = (double)rs.Object;
            //Console.WriteLine("Sum = " + sum);
            MessageBox.Show("Sum - " + sum);
        }
    }

#2

Hey Wolfie Wolf,

My name is Dash (@iamontheinet) and I am Developer Evangelist at Aerospike. Looking at your code, it seems like connection to the Aerospike server might be failing. I am referring to this line in your code AerospikeClient client = new AerospikeClient("192.168.2.11", 3000); If the connection is failing, client would be null and you’d get “Object reference not set to an instance of an object” error. I’d double-check IP and port settings of the server running your Aerospike DB instance and make sure it is accessible from your application.

I hope this helps.


#3

Hi Dash,

Many thanks for your prompt response… Using the same connection string I was able to update about 200K records and run queries against the DB. It’s only failing when I attempt to use the UDF…As a test, from a C# REPL I just issued:

#r “E:\Master\aerospike-client-csharp-master\aerospike-client-csharp-master\AerospikeClient\bin\x64\Debug IIS\AerospikeClient.dll” #r "system" using System; using System.IO; using System.Threading; using Aerospike.Client;

AerospikeClient client = new AerospikeClient(“192.168.2.11”, 3000);

Console.WriteLine(client); Aerospike.Client.AerospikeClient

It seems the value of client is not null. Do you have any other suggestions?

Many thanks

/W


#4

No worries. If you don’t mind, could you please surround your code in if..else as follows and see what happens? Notice that I am evaluating AerospikeClient.Connected property.

AerospikeClient client = new AerospikeClient("192.168.2.11", 3000);

// Check to see if the connection succeeded
if (client.Connected)
{
   Console.WriteLine("INFO: Connection to Aerospike server succeeded!");
   // Your code goes here...
} else {
   Console.WriteLine("ERROR: Connection to Aerospike server failed! Check your IP and Port settings.");
}

-Dash


#5

Hi Dash,

Sorry for the slow response, I’ve spent the day on this and I have some new information. In all of your example code the compilation is set for x64. If you change this to AnyCPU you can duplicate my crash. I promise you my client is connecting, I am able to run the exaples in the AerospikeDemo as well as the Training Visual Studio project you have available on GitHub. From what I can see the issue here has to do with some dependency being forced to X64. I just found this so I haven’t done any further troubleshooting yet. Can you confirm whether or not the c# client is able to be run in either 32bit or 64bit mode or if there is a requirement for 64 bit?

Thanks


#6

Hello!

To clarify, I wanted us to take a steb-by-step approach in understanding and resolving your problem and that is why I suggested evaluating AerospikeClient.Connected property. (Which, IMHO, is good practice so you can gracefully handle connection failures.) I was by no means undermining your abilities.

Alright, let’s get to it – you are correct in that platform dependency is in play here. Aerospike C# Client supports both 32-bit and 64-bit targets. However, it does not support AnyCPU target. The reason: our Lua interpreter is written in C and it is in fact C’s limitation bubbled up.

FYI, there’s another version of Aerospike C# Client called “AerospikeLite” that does not include/use Lua. It exists specifically for applications that do not have aggregation requirements. This “lite” version supports all targets including AnyCPU.

I hope this helps.

Tx! Dash


#7

Dude,

This was totally a 32bit vs 64 bit issue… I’m developing an application as a Visual Studio Package and, believe it or not, Visual Studio 2013 is a 32 bit app. Fortunately I’ll be using your cool server to do all the heavy lifting. Here’s what I learned and how I resolved it.

You’re using LuaInterface between the CLR and a C++ implementation of Lua 5.1.4. Unfortunately for me, and anyone else who would like to benefit form your technology, but is bound by 32 bit, that’s never going to work. First let me say that I am forward looking so I would never desire to lock code into 32 bit land, my preference would be, as I said before, to use AnyCPU, which I know is a .Net concept, but it makes sense as assemblies are compiled for both architectures. That being said Visual C++ doesn’t share the same concept but does allow the linker to be adjusted for different levels of verifiability (I am paraphrasing form this article - I actually don’t know jack about C++) hell I don’t know jack about Lua either!.. https://msdn.microsoft.com/en-us/library/31zwwc39 I tried to use /CLRIMAGETYLE: SAFE but the linker wasn’t having it because the Lua5.1Lib was compiled for IJW… I made a couple of attempts to swap out this lib with a 32 bit lib without success… the answer then arrived in the form of a transplant.

Kopilua is, “a C# port of Lua 5.1.4” and not dependent on any unmanaged code. It also has a port of the LuaInterface which is very close to your fork. KopiluaInterface uses all the same method signatures as your LuaInterface making it an (almost) drop in replacement; swap out the references and you’re almost there. There were a few discrepancies between methods, about five or six, but it was trivial to port the changes from your code into the Kopiluainterface. I’d be happy to share the upgraded version with your team if you’re interested.

The end result is that I am now able to compile the Aerospike Client, Demo and Training projects down to 32 bit and run the examples, specifically the aggregation examples, without any issues.

This, unfortunately, isn’t the end of the story as I still haven’t been able to make the example from the c# tutorial run. 64 bit or 32… There is definitely something missing from the tutorial. Now that this issue is resolved I’ll pull apart the examples in the Aerospike Demo and Training projects and boil them down to the least amount of code necessary to run an aggregation query from c#, and update you.

Guys, I think what you’ve done here is incredible and I totally appreciate how much effort has gone into this. It’s absolutely impossible for any one team to verify every possible use case before releasing. I’m thankful that you’ve open sourced this so that we, the community, can benefit and contribute, improving this awesome product to everyone’s mutual advantage. Of course if my project flies I’ll convert to a decent size customer actually, just don’t disqualify me for swapping out the LuaInterface.

Kind Regards

/W


#8

I just saw your response after I posted my solution. For clarity, from my understanding, the provided Aerospike c# client does not support 32 bit aggregation. The solution I provided resolves this.

No worries on your baby step approach, I understand, I’ve done my share of support; I just don’t have that kind of time.

/W


#9

I take it back about the example not working… it totally does. I just reset my Roslyn C# Interactive and ran the following right in the repl:

#r “E:\Master\aerospike-client-csharp-master\aerospike-client-csharp-master\AerospikeClient\bin\Debug\AerospikeClient.dll”

using System; using System.IO; using Aerospike.Client;

AerospikeClient client = new AerospikeClient(“192.168.2.67”, 3000);

string ns = “test”; string setName = “demoset”; string indexName = “aggindex”; string binName = “aggbin”; ResultSet rs = null; string luaDirectory = @“E:\”; LuaConfig.PackagePath = luaDirectory + @"?.lua"; string filename = “sum_example.lua”; string path = Path.Combine(luaDirectory, filename);

RegisterTask rt = client.Register(null, path, filename, Language.LUA); rt.Wait();

IndexTask it = client.CreateIndex(null, ns, setName, indexName, binName, IndexType.NUMERIC); it.Wait(); for (int i = 1; i <= 10; i++) { Key key = new Key(ns, setName, “key” + i); Bin bin = new Bin(binName, i); client.Put(null, key, bin); } Statement stmt = new Statement(); stmt.SetNamespace(ns); stmt.SetSetName(setName); stmt.SetBinNames(binName); stmt.SetFilters(Filter.Range(binName, 4, 7));

rs = client.QueryAggregate(null, stmt, “sum_example”, “sum_single_bin”, Value.Get(binName)); if (rs.Next()) { double sum = (double)rs.Object; Console.WriteLine("Sum = " + sum); } Sum = 44

Works fine!

Now to write some cool Lua code…


#10

I just wrote a simple aggregation app in C# VS 2013 that references Aerospike C# Client I downloaded from here that I then built specifically to target x86 (32-bit) platform.

This screenshot should give you an idea of what the console app does.

Selecting the first option (1) creates 10,000 dummy user records with randomly selected region (‘n’, ‘e’, ‘w’, ‘s’) and a tweet count of up to 20. As you can imagine, aggregation is based on these two bins.

Let me know if you want to check it out and I’ll publish it on GitHub.

Best, Dash


#11

Dash,

Yes, I would very much like to test that in my environment. No matter what I did I could not get past the null exception error whenever the resultset was called. I will test this in a fresh VM to ensure my setup is the same as yours.

Thanks

/W


#12

Didn’t you have to re-target the C# driver as well? From what I remember it’s default to x64?


#13

Here you go my friend: https://github.com/aerospike/aerospike-csharp-aggregation-x86

My dev env:

  • Mac OS X Yosemite
  • Visual Studio 2013 is on VMware Fusion instance running Windows 8
  • Aerospike Server is on VMware Fusion instance running Ubuntu 14.04

Hope this helps.


#14

Cheers mate… sorry for the slow response again… I’ll take a good look tomorrow… oh, and BTW. Found this in the debug logging output; my compliments to the chef, probably one of the best outputs of all time:

Jan 29 2015 02:20:19 GMT: INFO (as): (as.c::445) service ready: soon there will be cake!

/W


#15

Mate… again my apologies, I got bogged down with some other work and wasn’t able to look at your example yet, I promise to do so. Please don’t take this as a lack of interest; just business is business. I have a semi-related question and wondered if you would be so kind. You’ve been so helpful already I thought you might be able to shed some light on this for me.

I’m interested in understanding, from a technical level, exactly how the Aerospike Server instantiates Lua during the Aggregation process. I appreciate that I could go through the code, and it is my intention to do so, however I thought perhaps you would be able to quickly summarize it for me. The source of my question is interoperability. Lua is a versatile language and often used as “Glue”. From what I can see your client / server multithreading is incredible and not something that I would want to attempt to modify. I do, however, have some ideas about how to use Lua as a wrapper for the mono.evaluator C# compiler as a service. Ideally I would like to be able to wrap C# “scripts” in a simple lua wrapper script and throw them at the server as part of the aggregation process. I installed Mono 3.1.2 on the vagrant CentOS VM and using MonoLuaInterface and their provided lua.lua interpreter (that’s a lua shell written in lua) use the CLRPackage.lua to consume .NET assemblies. My goal is to re-configure Aerospike to use the .NET instance of LUA instead of the JITLua that you provided when processing the results of an Aggregation query. Is this something you would be able to give me advice on? I’d be happy to share the results of my efforts if you felt this was something the community could benefit from. I suppose I’ve already shared it; the idea and part of the solution is contained in this post.

Kind Regards

/W


#16

Never mind mate, I found it in aerospike-mod-lua-master.


#17

Hi Dash,

I finally had a chance to go through the Auerospike-csharp-aggregation-x86 example that you provided. Here are my comments. First, I performed this procedure on a fresh Windows 7 Virtual Machine running Visual Studio 2013 Update 4 against a new Vagrant install of Aerospike Server on OSX. So both VM’s are on OSX. In order to facilitate communication between the VM’s it was necessary to re-configure the Vagrant VM from port mapping to a bridged interface.

As per the instructions I used the “Download it form here” link, then clicked on the SDK link, which downloaded “aerospike-client-csharp-3.0.12.zip”

I moved this file to my projects directory and extracted it.

According to your GitHub documentation the procedure is to open AerospikeClient.sln. This doesn’t exist. The package includes Aerospike.sln and AerospikeLite.sln… and we know we don’t want lite because that doesn’t include aggregatoin. Within the AerispikeClient foldere there exists: AerospikeClient.csproj which is what I used.

Any attempt to change the platform from AerospieClient properties results in compilation for DebugIIS X64. In order to achieve 32 bit compilation it was necessary to use Configuration Manager to change the Active Solution Configuration to “Release” and then “Active Solution Platform” to “x86”. Following those steps I was able to compile for x86.

The next step suggested opening ASConsoleApp.sln but as before this doesn’t exist. The file is a Visual Studio Project, no a solution at this point (it won’t become a solution until after it has been loaded and re-saved). So I used ASConsoleApp.csproj.

The procedure suggests, “Add a reference to generated AerospikeClient.dll…” This could be misleading for some as your package already includes an orphaned reference to that assembly. It would be helpful to have either removed the existing reference before uploading the package or instructing to remove the existing reference before adding the new one as Visual Studio will not add a new reference to the same assembly if one already exists.

I did that (removed the old reference and added a new one to my newly generated x86 build) and ran the program. Option 1 worked as described. From aql on the server I can “select * from test.testusers” and return 10001 rows. Re-running the program and selection option 2, however, results in no results being returned.

Any thoughts?

/W


#18

Hey,

Thanks for your suggestions and catching couple of typos… I’ve updated README. As far as ASConsoleApp.sln is concerned, if you clone this repo - https://github.com/aerospike/aerospike-csharp-aggregation-x86 - it exists in the root folder. Which you can then open in Visual Studio 2013. (It worked for me.) But, opening the .csproj works as well.

Moving on to the real bug. In Program.cs, I had a typo in the bin name specified on this line:

stmt.SetFilters(Filter.Range("tweetCount", min, max));

It should have been tweetcount with lowercase ‘c’ like so:

stmt.SetFilters(Filter.Range("tweetcount", min, max));

I’ve corrected it in the repo and if you do the same in your app locally, it should work.

Let me know how it goes.

Thanks!


#19

Hi Dash,

Thanks for clearing that up; I can confirm that the example works now as expected. My main reason for coming back to it is the fact that I’m really stuck and not able to go any further with some pretty basic code. I was wondering if you might take a look and let me know what your thoughts are.

I’ve written, what I think, is a fairly simple Lua script:

local function aggregate_stats(map,rec)

map[‘Last’] = rec.Last map[‘TimeStamp’] = rec.TimeStamp map[‘TotalVolume’] = rec.TotalVolume map[‘Bid’] = rec.Bid map[‘Ask’] = rec.Ask map[‘TickId’] = rec.TickId map[‘LastSize’] = rec.LastSize map[‘BasisForLast’] = rec.BasisForLast map[‘TradeMarketCtr’] = rec.TradeMarketCtr map[‘TradeCondition’] = rec.TradeCondition map[‘i’] = map[‘i’] + 1 return map end

function stream_bin(stream) return stream : aggregate(map{i = 0, TimeStamp=0, Last=0, TotalVolume=0, Bid=0, Ask=0, TickId=0, LastSize=0, BasisForLast=0, TradeMarketCtr=0, TradeCondition=0},aggregate_stats) --: reduce(reduce_stats)

end

I’ve included all the fields that I may eventually want to return but for now I’m only using i, for the count, and last. Here’s the C# script.

#r “E:\Master\aerospike-client-csharp-master\aerospike-client-csharp-master\AerospikeClient\bin\Debug\AerospikeClient.dll”

using System; using System.IO; using Aerospike.Client;

AerospikeClient client = new AerospikeClient(“192.168.2.68”, 3000);

string luaDirectory = @“E:\”; LuaConfig.PackagePath = luaDirectory + @"?.lua"; string filename = “stream_example.lua”; string path = Path.Combine(luaDirectory, filename); RegisterTask rt = client.Register(null, path, filename, Language.LUA); rt.Wait();

string ns = “test”; string setName = “CL_TickSmall”; string indexName = “TradeMarketSmall”; string indexOnName = “TickId”; string computeOnName = “Last”;

IndexTask it = client.CreateIndex(null, ns, setName, indexName, indexOnName, IndexType.NUMERIC); it.Wait();

Statement stmt = new Statement(); stmt.SetNamespace(ns); stmt.SetSetName(setName); stmt.SetBinNames(computeOnName); stmt.SetFilters(Filter.Range(“TradeMarketCtr”,0, 36));

ResultSet rs = null; rs = client.QueryAggregate(null, stmt, “stream_example”, “stream_bin”, Value.Get(computeOnName)); if (rs.Next()) { Dictionary<object, object> result = (Dictionary<object, object>)rs.Object; Console.WriteLine("Last: " + BitConverter.Int64BitsToDouble((long)(result[“Last”]))); Console.WriteLine("i " + result[“i”]); }

And of course the database looks like this:

aql> select * from test.CL_TickTxt ±--------------------------±--------±---------±------------±--------±--------±----------±-------------±---------------±---------------+ | TimeStamp | Last | LastSize | TotalVolume | Bid | Ask | TickId | BasisForLast | TradeMarketCtr | TradeCondition | ±--------------------------±--------±---------±------------±--------±--------±----------±-------------±---------------±---------------+ | “2015-01-13 00:02:35.232” | “45.31” | “1” | “15880” | “45.30” | “45.31” | “3413790” | “C” | 36 | “01” | | “2015-01-13 00:04:11.504” | “45.30” | “1” | “15896” | “45.30” | “45.31” | “3414570” | “C” | 36 | “01” | | “2015-01-13 00:01:40.250” | “45.30” | “1” | “15876” | “45.29” | “45.30” | “3413650” | “C” | 36 | “01” | | “2015-01-13 00:03:35.559” | “45.30” | “1” | “15887” | “45.30” | “45.31” | “3414393” | “C” | 36 | “01” | | “2015-01-13 00:01:11.093” | “45.30” | “1” | “15868” | “45.29” | “45.30” | “3413611” | “O” | 36 | “4D” |

There are 58 records in the real table (I just copy pasted a few here for example). I realize that I have to do a filter as Aerospike needs to run a query before the aggregation but I’m attempting to filter on “TradeMarketCtr” which is always = 36 so that I get all the records. My assumption is that the aggregate function iterates all the records. It’s not clear from documentation if this is the case but if I use a condition to filter one specific record then the count is one. However with the above the count is always 10, not 58, which is what I would expect it to be. I’ve tried using filter.equal and filter.range, I’ve changed the value type for the TraderMarketCtr from Text to Int to long and nothing improves the situation. I’m I doing something really stupid here or is there something really broken?

Thanks… any advice you can give me will be major help. Clearing this issue will mean the different between continuing to attempt to use Aerospike or moving on to evaluate something else. This is potentially a very large project and I’ve spent a considerable amount of time writing an objective C to .Net integration piece. I would hate for that to have been for nothing.

Thanks


#20

Glad you got the github example working. Have you already compared your code with the github example? Sorry a little hard to eyeball and understand code on this page.

The other suggestion I have is that you first test your UDF/Lua script via aql. Here are a couple of links with more info on that:

Record UDFs - https://www.aerospike.com/docs/guide/record_udf.html

Aggregations - https://www.aerospike.com/docs/guide/aggregation.html