we had THP turned off if this helps in debugging.
cat /proc/meminfo
MemTotal: 99009560 kB
MemFree: 13786636 kB
MemAvailable: 18895604 kB
Buffers: 149912 kB
Cached: 6740264 kB
SwapCached: 0 kB
Active: 80052808 kB
Inactive: 4634552 kB
Active(anon): 78413012 kB
Inactive(anon): 378020 kB
Active(file): 1639796 kB
Inactive(file): 4256532 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 304 kB
Writeback: 0 kB
AnonPages: 77797316 kB
Mapped: 128076 kB
Shmem: 993848 kB
Slab: 103136 kB
SReclaimable: 78824 kB
SUnreclaim: 24312 kB
KernelStack: 9296 kB
PageTables: 179300 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49504780 kB
Committed_AS: 3953168 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 200540 kB
DirectMap2M: 6090752 kB
DirectMap1G: 96468992 kB
but min_free_kbytes are too low.
cat /proc/sys/vm/min_free_kbytes
67584
cat /proc/sys/vm/swappiness
60
looking at best practice doc, i do realise that these are not the best possible values. but imo it would help understanding what might be happening. Also we do use secondary indexes.