I’m working on setting up aerospike as a blob storage where each record is 2-4 MB in size. I have currently set it up as 2 separate namespaces - segments-disk (SSD backed) and segments-memory (memory backed) - where every record is written to both namespaces. The idea is to store the same data in segments-memory with a small ttl and in segments-disk with a large ttl. Currently, this is implemented as 2 separate Put operations (written in the go client library). Is there a way to copy a record from the segments-memory to the segments-disk namespace without having to send it over the wire twice? In production, we need to write 1.5 GB/s to the aerospike cluster so sending it over the wire twice increases the network egress to 3 GB/s which is expensive.
I’m using the go client library but examples in any client library will be helpful.
You could use data-in-memory true with your SSD storage option which will add a copy in memory (but with same TTL - you can’t do different TTLs between the two copies) and tied to the same Primary Index of the record. Reads will be faster from the memory copy, especially if your reads are long after the write and the record no longer in the post-write-queue in RAM. If you are reading immediately after writing, the reads can be served from the post-write-queue in RAM and you don’t even need the data-in-memory. That effectively gives you the short TTL in memory you are looking for. So some more specifics will help.
Also it isn’t guaranteed that the record will exist on the same node for all namespaces (e.g. migrations are ongoing or one of the namespaces may not be on all nodes, etc).