Hi, I have a 3 node cluster where I am seeing the following warning message in my logs almost every second. Jul 28 2017 14:45:04 GMT: WARNING (hlc): (hlc.c:564) HLC jumped by 1106 milliseconds with message from bb92f70c13e16fa. Current physical clock:1501253104839 Current HLC:1501253105695 Incoming…

Describe your environment you are rumning on.

Each node is a VM instance on an OpenStack cloud with 32GB RAM and 4 virtual CPUs. The Aerospike namespace is in-memory (not disk-backed) with 26GB memory allocated.

Your clocks are skewed between nodes. Do you have ntp running?

Yes, all 3 nodes have NTP running. The clocks are all in sync from what I can see.

It seems to be flooding the log file with these warnings (almost one warning log per second). Wondering if this affects the performance of the system?

Do the warnings show up on all nodes? Could you please paste the output of ntpstat from each of the machines.

Make sure that you are only synchronizing time using NTP in each VM, and not also trying to synchronize VM time to host time in the VM’s configurations. If you try to sync time using NTP and VM Host, any mismatch will cause the VM’s clock to jump back and forth.

will big latency between networks cause this HLC warning?

HLC jump warning in logs

pgupta July 28, 2017, 6:39pm 5

HLC - Hybrid Logical Clock - even though the skew is >1000ms between those two nodes, its only printing a “warning”, I believe the update is not rejected - so not an “error” per se.

Expiration value returned is invalid

Topic		Replies	Views
Aerospike Node Entering and Exiting the Cluster Frequently Configuration	9	2030	July 1, 2017
Aql client overwhelmed by "WARN AEROSPIKE_ERR_TIMEOUT" AQL	29	6274	February 8, 2017
Cluster node date delayed from the other 3 nodes	2	921	March 24, 2019
Node crash in AWS using the Aerospike AMI AWS ami , aws	8	5152	September 16, 2015
Aerospike cluster crashed after index creation Configuration aws	5	2894	September 22, 2015

HLC jump warning in logs

Related topics