Hi, I’m experimenting with gracefully shutting down nodes via quiesce and recluster.
I don’t have metrics yet, but anecdotally it seems like after issuing recluster, migrations take a lot longer than when a node rejoins the cluster.
Is this normal ?
I should mention that migrate-fill-delay is set to 300 seconds. Perhaps this should be higher if I’m quiescing and shutting down nodes ?
If you plan to take the quiesced node out permanently, then don’t delay fill migrations. (i.e. set migrate-fill-delay to 0).
If you plan to take the quiesced node out temporarily, you want to retain its data and bring it back (for e.g. rolling upgrade), then use migrate-fill-delay for the time you expect it to be out.
If you are bringing it back in less than 300 seconds then migrate-fill-delay = 300 is what you want.
Okay, that makes sense. In that case back to the original question: After quiescing this node, it took over 5000 seconds for migrations to clear up after issuing “recluster”. Normally when I restart a node it’s about 20 to 30 minutes for migrations to complete. Is this normal ?
It is hard to answer this without knowing all details of what you are transitioning from in terms of states.
If a cluster is fully stable, migrate-fill-delay is 0, you quiesce one node, recluster, what you have are fill migrations to new replica nodes which are starting empty. So all master records that are affected are copied over.
If you restart a node, which has existing data, then, in Enterprise Edition, we compare based on record metadata exchange (which is fast because metadata is small and is in memory), which records need to be migrated and only migrate those that need to be. (Rapid Rebalance feature). In this scenario, node restart migrations will be faster.
Fair point:
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.