Performance during nsup


#1

Synopsis

In some cases, it has been observed that the throughput and latencies take a hit whenever nsup runs on the namespace.

Scenario 1

Resolved

This has been addressed as part of AER-3250 in release 3.5.2 and 2.7.22.

Description

  • Storage Engine: data-in-memory with file storage
  • Number of objects: more than 200 million

In this case, when the nsup process runs, the latencies are affected. For example on a 8 CPU machine, it takes around 2 minutes to scan the entire namespace of 200 million objects. During this time, the percentage of transactions taking >8ms increase from 0.0% to 0.025%. This percent increase does not sound much but if the node is handling high traffic - say 175 KTPS, this translates to 3500 transactions taking more than 8ms. In this case, no actual evictions were taking place. So, the latency hit was purely due to the scan of index.

Workaround

One workaround is to increase the period when nsup is invoked using the “nsup-period” config parameter. The default period is 120 seconds. Increasing the period will mean that objects may not be expired at their precise expiry times.