Aerospike vs Redis for Incrementation of Ad Consumption

How Developers Are Using Aerospike Use Cases
1

Hi.

I was wondering if I should use Aerospike or Redis to store data for incrementing ad consumption (Amount of budget consumed of an advertisement’s total budget).

I’m working for a company developing an ad delivery system. We have a minimum traffic of 2,000 RPS, and at peek times this number can go up to 10,000; Ad consumption should be incremented for each of these requests. We have 10~20 web servers, and a single database must store all the consumption data, which is then read and incremented from the servers concurrently.

I fear the incremented requests would cause a bottleneck at peek times if we were to use Redis, since it’s not scalable. I’m assuming if we were to choose Aerospike, we wouldn’t have this issue since we can deal with increased traffic by adding new servers.

What I want to know is:

  1. How good is Aerospike’s incrementation? Why and How does it work? ( I’ve read Aerospike’s incrementation cannot be blocked or conflicted even with multiple requests ( increment counter · Issue #12 · aerospike/aerospike-client-go · GitHub ) )
  2. Are there any problems for storing and incrementing consumption if we were to use Aerospike.
  3. Are there any solutions for write (increment) block If we were to use Redis.

p.s For those with experience in developing adservers; what do you think of using Aerospike for storing ad consumption?

2

A specific ad should have a unique identifier in your application. That identifier would hash to a unique digest in Aerospike, its key, which is consistently hashed to a partition. There are 4096 partitions that are evenly distributed across the nodes of the cluster. When you add nodes to the cluster the partitions get automatically redistributed, but the client inside your application still knows where the record is, even if its partition is now on a different node. This is how Aerospike scales horizontally without you needing to adjust things on your application-side.

Increments are atomic operations. You do not need to read the record, increment it at your application, then write it back. The record is locked, the value in the bin you specified modified, and the record is unlocked. This is a very fast operation.

While the lock is held, other operations on that record will queue up. If the queue is exceeded you will see an error 14 Key Busy. If you see such a symptom, the size of that queue can be dynamically modified. You could also partition your key into several sub keys to spread the load, and use a batch-read to get their combined value. Finally, you could view a hot key as a signal that something malicious is happening, such as a bot hitting your ad faster than humans could. You could then use a blacklist to hold away such a device for a given time.

The storage engine will determine the speed the counter can be accessed and the rate it’s incremented. Counters are typically part of an in-memory namespace, but they can also be part of a larger record that lives on SSD. There is a third storage mode data-in-index that is ideal for cases where a counter is a record with a single integer. All 8B of the counter can fit directly in the primary index entry and require no extra storage space. This is also the fastest possible counter for you to have.

3

I greatly appreciate your reply. I now understand Aerospike’s consistent hashing and automatically distributing features. Could I ask you some more questions below?

  • Do you mean I wouldn’t need to manually change sizes for the queues? (Would I not be able to manually set the queue sizes?)
  • Is the partitioning of sub keys done automatically? How would I make a sub key in actual practice? I’d like some more information.
  • Do you think it’s better to use Aerospike rather than Redis for my use case? I would love to hear your opinion.

p.s Continuously for those with experience in developing adservers; what do you think of using Aerospike for storing ad consumption?

4

  1. Yes, you would manually change the transaction-pending-limit using asinfo or any other means of issuing an info command. You could also do it from the client you’re using in your app. This is a system-wide setting, however.

  2. No, you would model it differently. If the key was originally ‘a123’ you would split the identifier in your application to ‘a123-1’, ‘a123-2’,‘a123-3’, …, ‘a123-N’. You can now assign each appserver or each process a target of 1-N. This could be manually or (truly) random. The keys are now incremented in parallel, reducing the load on each portion. When you want to read the counter you would issue a batch-read for all the sub keys, and add them up on your application side.

  3. The majority of the adtech space does. Specifically for this use case, it depends on performance and scale. Aerospike is a real distributed database, and Redis is not. If your scale fits in a single machine then Redis is probably more than enough for you. If you see scaling in your future, you probably want to start with Aerospike, because it’ll be doing that far better. Should I choose your ice cream flavors too? :slight_smile:

5

Perfect! That’s everything I need to know!

I’d have to try the flavors myself to ask anymore questions :wink: