There are some highlighted items from this post:
Why am I growing data on the remote clusters so fast, and not evenly?
From the Support email the inequality of records was determined to be AWS1 was added earlier and AWS2 added later.
For this point:
- change the default-ttl from 1095D to 15D (1095 was from the config on our local cluster and should have been edited for the remote cluster)
The configuration parameter
default-ttlonly impacts new writes, and that is limited to new records not being written with explicit ttl values. Existing records will not be impacted through modification of this configuration parameter, thus your existing records would still be expiring at 1095D and not expiring as hoped at 15D.
The solution is to effect a Scan touch operation and update the TTL This can be accomplished using a User-Defined Function (UDF) with the Lua Programming Language or programmatically(using our supported Client Libraries).
To accomplish this using the Lua Programming Language see: Developing UDF Modules | Developer
The following article contains a verify basic example of how to update the ttl see: How to 'touch' record on UDF
NOTE: Make sure you do not enable
xdr-ship-bins(http://www.aerospike.com/docs/reference/configuration#xdr-ship-bins) as this would then only ship the changed bins.
This document outlines the process to execute the UDF: http://www.aerospike.com/docs/tools/aql/udf_management.html
In the case of needing to kill_scan more details can be found in the AQL docs under Query and Scan Management see: http://www.aerospike.com/docs/tools/aql/query_scan_management.html
To deepen your understanding on UDF see: User-Defined Functions (UDF) Development Guide | Developer
Scan touch can also be performed programmatically, using our supported Client Libraries
For Example, the Java Client API allows you to perform operations on records. http://www.aerospike.com/docs/client/java/usage/kvs/multiops.html Here is an example of Scan Records code http://www.aerospike.com/docs/client/java/usage/scan/scan.html Here is the API Reference including information on the touch method. aerospike-client 7.2.0 javadoc (com.aerospike)
Here is another example using C Client API on Touching a Record http://www.aerospike.com/docs/client/c/usage/kvs/multiops.html#touching-a-record
Can I do something to change the widths of the buckets on the remote clusters to reflect the different ttl?
Histograms are configurable at the namespace level and not user configurable on a cluster to cluster basis.