Why insert/put within loop insert only 1 row?

Client Libraries Node.js Client
1

I have been trying to put data of about 100 rows in foreach loop in node js…

Unfortunately I didn’t find any solution… It just insert 1 row and, print error message: AerospikeError: Client timeout: iterations=1 lastNode=localhost:3000

There must be some problem here because it just insert 1 row… I couldn’t find solution for this simple problem…

Here is my node js code:: array values gets from mongo db;

const allEvents = await Event.find();
        const key = new Aerospike.Key('evDb', 'allEvents', 'PK');
        for(var i=0; i < allEvents.length; i++){
            await client.put(key, {
                PK: allEvents[i]._id.toString(),    
                eventContent: {...allEvents[i]},
            });
            if(error){
                console.log(error);
            }
        }

Remember, it insert 1 row, so data is correct as per I have checked…

It’s very difficult for beginners as they don’t have enough demonstration of such…

Your suggestions will help me save several hours of unfruitful work hours

thank you

2

I don’t know much about the Node.js client but the error indicates the transaction timed out. Did you try (just to troubleshoot) writing two records individually… could it be the problem is in the way elements are consumed from ‘allEvents’ and fed into the Aerospike client?

3

Oh, I checked with a colleague of mine who mentioned that the key is a constant, so that would explain the single record if all the updates are done under the same key (you probably would ant to update the key with the event’s ID on every iteration). Not sure what the timeout error is or where it comes from but it is also not clear from the code where this error variable is getting set. Hope this helps.

4

@meher Thank you for the response. We did figured out most things. AS is really good in comparison to others key-value memory database. But what I feel is missing in AS is proper documentation on sort write policy. As we figured, by default sort is unordered, but for any web applications ordered sort – it’s one of the primary features that needs to be look at. I think it’s probably there in java, but we still couldn’t figure out in node.js even while we use UDF along. If you can please point out relevant doc in node.js KEY_ORDRED write policy in node.js , we did tried few things but never worked. We are in to enterprise version and mostly our data is in scalar type. Thank you so much for your help Meher…

5

The documentation can definitely be improved, especially for some of the clients and we are working on improving on this front. Stay tuned…

Regarding your question I found the following snippet on our sandbox after selecting Node.js:

// Initialize a write policy
policy = new Aerospike.WritePolicy({
    key: Aerospike.policy.key.SEND
});

;(async () => { 
    try{
        // Connect to the client
        client = await Aerospike.connect(config);
    
        // Write the record
        await client.put(key, bins, [], policy);

        // Create and set map policy
        const mapPolicy = new Aerospike.MapPolicy({order: maps.order.KEY_ORDERED});
        let ops = [maps.setPolicy('report', mapPolicy)];

        // Apply the map policy
        await client.operate(key, ops)
        
        let record = await client.get(key);
        console.info("Create succeeded\nKey: %o\nRecord: %o\n", record.key.key, record.bins);
    }
    catch(error){
        console.error(error);
        process.exit(1)
    }
    finally{
        if(client) client.close()
    }

Hope this helps.

6

Thank you @meher for your swift response. It really helps! However, not the exact thing I was looking for – nonetheless, it is of great help in many ways… May be, I couldn’t explained better in the above comment.

Here is what exactly I am looking for your suggestions: A simple scalar data type set (like typical Relational databases). Please look at this set format:

+------------+------+-----------+--------------+-----------+
| pk         | sort | userId    | played       | contestId |
+------------+------+-----------+--------------+-----------+
| 1658851106 | 4    | 100000052 | "Basketball" | 1500012   |
| 1658851283 | 9    | 100000050 | "Cricket"    | 1500017   |
| 1658851006 | 2    | 100000051 | "Football"   | 1500009   |
| 1658851058 | 3    | 100000052 | "Football"   | 1500011   |
| 1658851229 | 7    | 100000050 | "Cricket"    | 1500015   |
| 1658851171 | 6    | 100000052 | "Basketball" | 1500014   |
| 1658851151 | 5    | 100000052 | "Basketball" | 1500013   |
| 1658851250 | 8    | 100000050 | "Cricket"    | 1500016   |
| 1658850954 | 1    | 100000050 | "Cricket"    | 1500000   |
| 1658851315 | 10   | 100000051 | "Tennis"     | 1500018   |
+------------+------+-----------+--------------+-----------+

Node js put query:

 const pKey = Math.floor(+new Date() / 1000); //instance epoc timestamp 
 const sort = define.sort; //for sort count we maintained exclusively different namespace which is incremented everytime new records are added to avoid UDF or complex query
 const userId = defined.userId; //Secondary Index
 const sportName = defined.sportName; //Secondary Index
 const contestId = defined.contestId;
 const key = new Aerospike.Key('test', 'playedContests', pKey)
 
 let bins = {
   pk: pKey,
   sort: sort,    //sort basically is used to limit records for pagnation
   userId: userId,
   played: sportName,
   contestId: contestId
 }
   // Write the record
 await client.put(key, bins);

Node js get query with filter and sort as per defined sort value (limit range) we have in place:

    let query = client.query("test", "playedContests");
    query.where(Aerospike.filter.equal("played", "Cricket"));
    query.where(Aerospike.filter.range('sort',1, 3))
    var stream = query.foreach();
    stream.on('data', function (record) {
            console.log("Response :: ", record)
      //process record
    })

This query will only return me 1 record as per in the records table above because sort bin will read only between 1-3 sort (limit range) values, but when you see we have about 4 records where bin (played = cricket)… In large records it’s impossible to filter and then limit with value range (sort bin here) with this logic.

My questions are:

  1. can we use rank range here? (but here also I am talking about simple scalar data not list or map scan).
  2. Is there any ordered insert operation for such type of simple scalar data like order record data in sequence as per insertion timestamp ASC order of PK?
  3. Do you have better suggestions to deal with the data structure to met such requirements?
  4. 5 bins with 2 secondary index OR 2 bins with 1 map/list data… which is better performant for 70:30 read:write data…

Thank you so much for all the support and suggestions.

7

Glad that some of the answered helped a bit at least. Coming to your questions, let me try, based on my limited knowledge around data modeling:

  1. It seems you are looking at doing the union of two queries, which is something that I don’t think is supported.

  2. The Aerospike database does have an internal metadata (called the last update time or LUT) for the timestamp the record was last updated (or inserted) but I don’t think it can be used for sorting on queries… It can accessed in UDFs (User Defined Functions) or filters but not for sorting as far as I know. Having said that, our most recent release does support pagination for secondary index queries (the order will be by ‘digest’ on a per partition basis), but I am not sure if all is supported on the node.js client.

  3. / 4. I am not an expert on this front but usually the first answer is ‘it depends’, and based on factors like SLAs, the infrastructure details and storage configuration… but it seems that for any sorting type capabilities, you would have to go with map/list anyways…

8

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.