Hi, I have a 3 node cluster where I am seeing the following warning message in my logs almost every second.
Jul 28 2017 14:45:04 GMT: WARNING (hlc): (hlc.c:564) HLC jumped by 1106 milliseconds with message from bb92f70c13e16fa. Current physical clock:1501253104839 Current HLC:1501253105695 Incoming HLC:1501253105945 Tolerable skew:1000 ms
Would any one here have any idea how to fix this error?
Thanks
Describe your environment you are rumning on.
Each node is a VM instance on an OpenStack cloud with 32GB RAM and 4 virtual CPUs. The Aerospike namespace is in-memory (not disk-backed) with 26GB memory allocated.
Your clocks are skewed between nodes. Do you have ntp running?
HLC - Hybrid Logical Clock - even though the skew is >1000ms between those two nodes, its only printing a “warning”, I believe the update is not rejected - so not an “error” per se.
Yes, all 3 nodes have NTP running. The clocks are all in sync from what I can see.
It seems to be flooding the log file with these warnings (almost one warning log per second). Wondering if this affects the performance of the system?
Do the warnings show up on all nodes? Could you please paste the output of ntpstat from each of the machines.
Make sure that you are only synchronizing time using NTP in each VM, and not also trying to synchronize VM time to host time in the VM’s configurations.
If you try to sync time using NTP and VM Host, any mismatch will cause the VM’s clock to jump back and forth.