Normal operation of READ and WRITE is standard and node .103 is not slow
Only looking at the NSUP logs :
Mar 07 2022 18:28:54 GMT: INFO (nsup): (thr_ nsup.c :1174) {buck_xxxxxxx} nsup start
Mar 07 2022 20:32:20 GMT: INFO (nsup): (thr_ nsup.c :1098) {buck_xxxxxx} Records: 41115293, 0 0-vt, 40897418(118769533) expired, 0(0) evicted, 0(0) set deletes. Evict ttl: 0. Waits: 0,0,6889892. Total time: 7405785 ms
The process takes more than 2 hours and other nodes are able to complete the process in more batches (shorts and with less expired) but overall keeping the node cleaner
What is the meaning of this? Waits: 0,0,6889892
Does anyone have a similar case? We think it could be an SSD problem but it is strange we don’t see any effect on the write/read performance
That is a pretty old version and the whole nsup sub-system was since redone, but if I remember correctly, one of the typical scenario for nsup to fall behind was when the generated delete transactions were not processed fast enough. So that wouldn’t be an SSD problem but a connectivity issue or even just network. The meaning of the Waits is documented here (you need to click on ‘Show removed server messages’):
Here is the relevant section, and that number is actually the throttling because of reaching the max 10,000 delete transactions. I think that it would be milliseconds rather than microseconds and that would pretty much explain that high total time (the fact that the message that replaced this one, between versions 3.14 and 4.5.1 has the waits in millisecond is another potential evidence that it is milliseconds…).
Waits: Accumulated waiting time for different stages of delete to finish, in microseconds. In each cycle, nsup performs set-deletes before general expiration and eviction.
n_set_waits: The first wait is the number of microseconds that nsup slept during set-deletes stage while waiting for the nsup-delete-queue to drop to 10,000 elements or less (Throttling).
n_clear_waits: The second wait is the number of microseconds until the nsup-delete-queue cleared (including the previous namespace if applicable) before beginning general expiration and eviction (Minimize unnecessary eviction if deletes already pending). For the last namespace in the nsup cycle, this is reported on its own line, nsup clear waits: 1441
n_general_waits: The third wait is the number of microseconds nsup slept during general expiration and eviction while waiting for the nsup-delete-queue to drop to 10,000 elements or less (Throttling).
