We added a new node (to an initial cluster of 1 and migrations went fine. However the secondary indexes on the newly added node lag behind the first one. Is this expected? Shouldn’t we expect the secondary index statistics to match up on both nodes?
Does this mean that if node #1 goes down, node #2 will rebuild its secondary indexes?
Could you share your secondary index stats?
Server version may be helpful.
v3.14.0
root@aerospike-0:/# asinfo -v "sindex/exports/idx_plantaindv2_account" -l
keys=1322
entries=2809812
ibtr_memory_used=80384
nbtr_memory_used=92151484
si_accounted_memory=92231868
load_pct=100
loadtime=698786
write_success=4931776
write_error=0
delete_success=2121925
delete_error=0
stat_gc_recs=5422
stat_gc_time=24
query_reqs=13
query_avg_rec_count=121774
query_avg_record_size=746
query_agg=0
query_agg_avg_rec_count=0
query_agg_avg_record_size=0
query_lookups=13
query_lookup_avg_rec_count=121774
query_lookup_avg_record_size=746
histogram=false
root@aerospike-1:/# asinfo -v "sindex/exports/idx_plantaindv2_account" -l
keys=198
entries=34492
ibtr_memory_used=18688
nbtr_memory_used=1121896
si_accounted_memory=1140584
load_pct=100
loadtime=79
write_success=437459
write_error=0
delete_success=418136
delete_error=0
stat_gc_recs=8
stat_gc_time=0
query_reqs=13
query_avg_rec_count=7036
query_avg_record_size=741
query_agg=0
query_agg_avg_rec_count=0
query_agg_avg_record_size=0
query_lookups=13
query_lookup_avg_rec_count=7036
query_lookup_avg_record_size=741
histogram=false
Is there a way to trigger a rebuild of the secondary indexes on the second node? Or is this the expected behaviour?
Also if node 1 now goes down, will node 2 rebuild all the secondary indexes?
It should automatically balance out. In some cases, dropping and re-creating the index is the best way to handle things like this (which will make it re-build). I’m not sure about your particular problem though…
There is a known issue on 3.14.0 that impacts secondary index:
If this is the issue you encountered, as a workaround, on version 3.14.0, you could restart the node that has less keys/entries and that should force the sindex to be rebuilt.
Secondary index are built on both master and prole transactions… so nodes would be able to take over server secondary index queries when a node leaves the cluster.
The bug caused records ‘created’ through an incoming migration to not have their secondary index created. So if you added an empty node, most records on that node would be created through incoming migrations and would be missing their secondary index.
Upgrade to version 3.14.0.2 and you should be fine, based on your description and the data you shared I am fairly certain this is the issue you did hit.
Aah, thanks Meher. That explains everything.