We are looking ways to store list of objects in aerospike using aerospike php client.
I can use simple set method and pass list so data getting stored using native serializer and getting retrived effectively.
We are storing client wise users data in cache to effectively use those objects in number of placess.
We have pages where we want to display almost all user details based on different use cases.
Our key here is clientId and value will be list of user objects for that client.
User object contains information like username, email, mobileno, roles etc.
Now problem is in some clients they have more than 1000 to 2000 users.
When storing this many objects its crossing more than 1MB memory. How to handle these cases in aerospike ?
Previously we are using memcache and redis and are able to do same.
In aerospike we are using SSD backed persistent memory and configurred write_block_size as 1Mb.
On some request due to write block size is more than 1MB aerospike is rejecting same.
The 1Mb limit does not apply if you configure your namespace for storage in RAM only. If you want persistent storage, 1Mb is a hard limit. And that 1 Mb does include a fraction overhead as well.
Can you come up with a composite key - and group your data in 1 Mb chunks. For example:
Your key: “user1:geodata” → one set of bins (record), “user1:billingdata” → another set of bins (record)… you get the idea.
Or key = “client1” bin = number of usersets - one record- then key= “client1:userset1”, “client1:userset2” … has all the data and based on number or usersets, you know how many composite keys you have traverse in your application.
Another way: first set is always: “client1:userset1”, first bin is always - int hasNext (1 or 0), then user data. If hasNext = 1, further traverse “client1:userset2” – which has same hasNext bin also before storing userData.
I didn’t understand your answer completely , before understanding this is write_block_size limit is per key or its per bin .
Menas i have key called clientkey and its value size of clientkey is 1 MB or each bin size is of 1MB ? If It is Bin then definitely i can store my data in multiple bins . As per my understanding size limitation is on value of Key . is this correct ?
As per above answer you are suggesting to separate my list of objects into multiple keys or multiple bins with same key ?
RAM storage → no write-block – record size limit does not apply.
SSD / HDD (file) storage case:
write-block = 1 Mb (max 1Mb- you can set it lower, 128KB recommended for SSDs)
Comprises as many records that can be fit into it. => records cannot exceed write-block size
write-block in RAM is filled with incoming records, when full or certain time passes, written to SSD / HDD.
Aerospike Record = key + one or more bins, bins can contain integer/double/string/bytes/lists/maps… etc.
key = using the term loosely but effectively here to give you a general idea. You must read the link below to fully understand Aerospike Data Model.
[ http://www.aerospike.com/docs/architecture/data-model.html ]
BTW, if you have the budget, explore http://www.aerospike.com/schedule/ - two day virtual training, full immersion, hands on. Seems like you are at the right stage to benefit from it.