Implement redis style lpop or blpop

How Developers Are Using Aerospike
1

I am trying Aerospike for the first time.

I need to implement a queue manager. These are strings ( of mails ) that need to be processed. The sizes of these strings may vary hugely ( 1k to 5Mb ) . Typically there would be 10 to 100 mails every second to process.

There will be multiple enqueuers and multiple dequeuers so I plan to push all these strings into a queue and then dequeue them using a lpop in the dequeuers

Having a redis is not a good option because the memory consumed will be huge for large strings.

What can I do to implement lpop in Aerospike

2

Ram,

Aerospike have LargeList which you could use to enqueue and dequeue the element into. Please check out

But for your need there is nothing like BLPOP (redis similar) in aerospike, you will need to take care of this is application by may be say polling …

Given that you are building it over KVS, if there is only 1 queue, system may not scale well … so you may want to do circus of split it out into multiple queue and use round robin polling to work with.

– R

3

Thanks Raj If I cant implement blpop but only lpop that should be fine. I could code my dequeuer to keep polling

I am not pretty sure on multiple queues , If I were to have a busy system I would probably run many dequeuers. Having a single queue is never concern as long as I can scale my dequeuers. What is the advantage of using multiple queues ?

4

On a single record all the enqueue and dequeue operation is going to be serialized … slowing the processing down … .

– R