How to manage traffic spike for single data item?

Operations Planning
1

Hi, I am going through the aerospike documentation and it is an interesting implementation. Almost everything is handled for a internet scale performance.

Let us say I implement it for an ecommerce site, and offer an unbelievable deal on a product. I will get millions of hits for that product.

I understand that aerospike distributes data on multiple nodes, so traffic for different products would be distributed to different nodes. But how to scale for increased load for a single product?

regards

2 Likes
2

This is not actually a noob question. This is wonderful question which I think lot of e-commerce people will be trying to solve. Let me take my dig at addressing the problem.

I dont think this will be solved by any data store with very less copies of the data. typically nosql databases have 2 or 3 copies. the reads can be scaled a bit by doing round-robin between copies of the record. but for writes all of the them have to go to same node which can be a hotspot. For objects which do not have writes or writes are very rare, a cdn type of caching or application level caching where many copies of the data are created is the way to go.

If you still want to solve it using Aerospike, one idea could be… For one product, create multiple physical records and logically split the entity. Say, you want to sell 100,000 items of a product with key p, you can split them into 50 items with key p01-p50 and store the inventory as 2000 each When the user’s query hits, randomly choose one of the 50 keys from p01-p50 and let him buy. To give a consistent view of the product, a logic can be chosen based on the sessionid (for e.g. modulo 50 of sessionid) or some other parameter. This logic will work if it is not necessary to be too fair to the users as there is some form of randomness in this logic.

Let us know your thoughts about the above solution. Its not straightforward but could be a good workaround. If you have some good ideas, please let us know too.

3 Likes
3

Thanks Sunil. Just wanted to know if AS had this built in and I am not able to figure out if this is available out of the box.

Application managed caching, on another layer of Redis servers seems to be the way. When launching a deal, items anticipated for high load can be pre-populated in the cache. Along with cache invalidation hook to ensure they are updated on all copies when something changes for that product.

Can also be automated to bring up items to this cache, which are accessed on a rate higher than a threshold - when you can’t predict what external factors could trigger an increased interest in a product.

Edit - Key eviction | Redis - LRU with Redis. And then distribute this cache on multiple instances.

1 Like
4

This is not built into the aerospike. This needs to be done by the application.

Yes, a redis or a memcache layer above can be used to not hit aerospike always. But you still have to maintain multiple copies of the data without with redis/memcahce also may not be able to take this much of load for 1 key. Automating this based on the hotness sound a good idea to me.

1 Like
5

Hi @sunil,

The C client offers an option of to randomize reads from primary and replica nodes.

From https://www.aerospike.com/docs/client/c/usage/consistency.html

Select Reading of an Arbitrary Replica

By default, all client reads will be directed to the master replica (policy value AS_POLICY_REPLICA_MASTER.) In some cases, however, it may be desirable to spread the reads over all available replicas. (For example, the performance impact of reading a hot key may this way be reduced by on the order of the replication factor.) In this case, the application may select reading from an arbitrary (“random”) replica by setting the read.replica policy to AS_POLICY_REPLICA_ANY:

p_policies->read.replica = AS_POLICY_REPLICA_ANY;

Any chance this could be provided for in the Java client as well?

1 Like
6

Hi @sunil,

Handling hot keys by reading from replica is a good feature to have, it would solve a lot of use cases especially for e-commerce sites. Also as this is supported in C client as mentioned by @Abhi, i think it ll be great if you could prioritize this feature requests.

7

Hi Abhi,

Thank you for your feature request to provide this option in the Java client. It is under strong consideration. Please stay tuned.

On a general note, we announce all new server and client releases via our Announcements page, so feel free to bookmark this page to get notified of new features and fixes.

Best regards,

Maud

8

Distributing reads across proles is on the java client roadmap, but it’s not the panacea you might perceive it to be. Here are some reasons why:

  1. Only reads could possibly benefit. Writes must always go to master, even for hot keys.

  2. The read-modify-write paradigm can result in more generation errors if prole reads are allowed. If a key is read from a prole while simultaneously being updated by some other thread/client, the generation could change on master before prole. Therefore, the final write could fail more often on the generation check. If the key is read from master, the race condition between master/prole is eliminated.

  3. Prole reads require the client to request/process all replica maps for each node. This would increase the amount of data cached by the client by a multiple of the replication factor. Processing time in the cluster tend thread and key partition retrieval is also increased. Not a deal breaker, but something to keep in mind.

  4. There will be no benefit if the number of hot keys approaches the number of nodes in the cluster. Suppose a four node cluster with replication factor of 2 is deployed. If there are 4 (or more) products that are very hot, the distribution would look something like this:

    hot : master prole
    p1  : n1     n2
    p2  : n2     n3
    p3  : n3     n4
    p4  : n4     n1

The distribution of p1 off n1 to n2 is negated by equally hot p4 which adds to n1 load. Nodes are being swapped, but each node’s read load remains the same.

The only time prole reads provide any benefit is when one key is substantially hotter than any other key in the cluster.

9

Java client 3.1.1 has been released which allows for reads to be distributed across proles. See “ClientPolicy.requestProleReplicas” and “Policy.replica”. This functionality is disabled by default and can be enabled by users if using Aerospike server versions >= 3.5.9 (not released yet).

http://www.aerospike.com/download/client/java/notes.html#3.1.1

3 Likes