Initially we did tests with Aerospike. In one test we have had 4 Aerospike nodes with 80% of disk filled in (bcache with same size backing disk). We heavy load Aerospike nodes by read operations. And add 5th node. The goal was to see how Aerospike behave during re-balancing. We found significant performance degradation and plus read operations from EBS. The last was surprise for us, because we expect complete caching. Before re-balancing we did not have any read operation, but we query just subset of data, and it was completely cached. But looks like re-balancing start reading whole data and produce read operations on EBS.
So, after it, we did deep investigation on bcache without Aerospike. Found that we are able to completely avoid read operation only with 75% backing disk capacity. Even in this case to cache whole data we need to read all data several times (3-4). Each operation reduce amount of reads. And on 4-5 time they are completely gone.
RAID-1 promise to resolve all this problems. It would be not a cache, but complete plain mirror. md kernel driver can be configured to read from EBS only if SSD drive is unavailable. In other case all read operation go to SSD, which is good. Unfortunately all modern kernels have a bug in this functionality. Only latest kernel have fix, so we need to wait while fix will be available in Ubuntu (which we use for our production).