Uneven memory usage distribution

[totola]

in which circumstances , the partition would store more keys but used less memory . we had a cluster with three nodes , but there is a huge memory usage difference

how does this happen and how to avoid it （community version ,not enterprise）

[meher]

Do you have secondary indices defined by any chance? Could it be a case of memory not garbage collected on the secondary indices and the node which is using much less memory may have been added after the other ones or recently restarted?

[totola]

the node which use less memory is added recently , but after that , we add another two new nodes yestoday, and the new two nodes looks good

and we do not use secondery index , is there any other possibilities for this kind of issue ?

thank you very much

[meher]

So, next step would be to confirm where the memory is used. I assume your namespace is configured with data-in-memory true, right? We can first confirm how the memory is distributed, to make sure nothing unexpected.

What does the following command show (to be run from asadm)

Admin> show stat like memory_used
~test Namespace Statistics (2020-08-25 04:06:38 UTC)~
NODE                    :   d7af462a411d:3000
memory_used_bytes       :  0
memory_used_data_bytes  :   0
memory_used_index_bytes :  0
memory_used_sindex_bytes:  0

But actually since the disk usage seems similar, could it be that this one node is simply configured with data-in-memory false and the other ones are configured with data-in-memory true?

[meher]

I think this last theory should be the right one. I hope so… since ~14 GiB on disk would likely be ~8-9 GiB in memory which is the gap on that node that is using less memory.

[totola]

you are awsome , thank you very much . it does misconfigured cause that issue , I lost the data-in-memory setting.

these issue borther me a long time ,thank you very very much for your kindness and patience

All the best wishes for you : )