Can I send a CVS file to a UDF to insert its contents into the DB?


#1

I have a large CSV file where I want to insert each line as a different record (maybe 30,000 lines) and concatenating two of the fields within the CSV as the key. If there is an existing key I want to append the information to the existing record.

My intent is to pass the CSV file to the UDF and have the processing done on the DB side - is this possible?

Is this a scenario of a multiple record operation which is a limitation for Aerospike?

Thanks


#2

Hi Bret

You can use a UDF to process the CSV file. But there is a java-based CSV loader at: https://github.com/aerospike/aerospike-loader

It would be very simple to modify it to do what you need, and maybe use a UDF to do the append.

Hope this helps!


#3

Hi Thanks for the reply. Unfortunately, that code we were already using and found that by using the client side we code to upload the CSV it meant that we were writing 2.1 million times to the database (each line of the csv file). This has increased the write time to over a minute.

That’s why I was hoping that we could some how send the csv to UDF.

But there seems to be no way to create a key on the UDF side only from within the client - is that correct? Is it possible to create a key using Lua?

Thanks Bret