Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clock synchronization is a nightmare

arpitbhayani.me

Post reply on HN
Print view View original

141–150 of 165 posts

Re: Clock synchronization is a nightmare

#141

Post by otterley »

sreekanth850 wrote: post #128
otterley wrote: post #126

Earlier quoted context omitted.

You can’t be certain that any given mutating operation you perform now won’t be relied upon for some future operation, unless the two operations are performed in entirely different domains of data. Even “not touching (by which I assume you mean mutating) the same data” isn’t enough. If I update A in thread 0 from 1 to 2, then I update B in thread 1 to the value of A+1, then the value of B could end up being 2 or 3, d…

In distributed systems, dependencies flow forward, not backward. Causal dependency only exists when an operation actually references earlier state. If B = A+1, then yes, B is causally dependent on A and they must share an order. But that dependency is created by the application logic, not assumed globally in advance. We shouldn’t impose a universal timeline just because some future operation might depend on some past…

> But that dependency is created by the application logic, not assumed globally in advance.

Not necessarily. If you allow users to perform arbitrary operations on the data, all bets are off.

> We shouldn’t impose a universal timeline just because some future operation might depend on some past one. Dependencies should be explicit and local: if two operations interact, they share a causal scope; if they don’t, they shouldn’t pay the cost of coordination.

Application developers can already avoid that cost by using separate databases for different data domains. That’s very explicit and effectuates intent rather nicely.

Re: Clock synchronization is a nightmare

#142

Post by nickpsecurity »

dmazin wrote: post #73

I highly recommend anyone to look up how PTP works and how it compares to NTP. Clock sync is very interesting. When I joined an HFT company, first thing I did was understand this stuff. We care about it a lot[1]. If you want a specific question to answer, answer this: why does PTP need hardware timestamping to achieve high precision (where the network card itself assigns timestamps to packets, rather than having the…

The Wikipedia article has a good summary:

https://en.wikipedia.org/wiki/White_Rabbit_Project

Re: Clock synchronization is a nightmare

#143

Post by jeffbee »

Dylan16807 wrote: post #113
jeffbee wrote: post #52

Earlier quoted context omitted.

Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. The control loop for host PTP benefits from regularity. Anyway NICs that support PTP hardware timestamping may also use PCI LTR (latency tolerance reporting) to instruct the host operating system to disable high-exit-latency sleep features, and popular operating systems respect that.

> The control loop for host PTP benefits from regularity. How much regularity? If you sent PTP packets with 5 milliseconds of randomness in the scheduling, does that cause real problems? It's still going to have an accurate timestamp. > instruct the host operating system to disable high-exit-latency sleep features Why, though? You didn't explain this. As long as the packet got timestamped when it arrived, the CPU can…

> PTP packets with 5 milliseconds of randomness in the scheduling

This should not matter, unless you are a 5G telecom operator running at a high frequency. Gaussian noise in the master is not important to PTP. Being a master is easier than being a slave.

If you are running PTP at 128 Hz like a telecom, delays that large might lead to slaves resetting their state machines, which would blow the whole thing up.

> The CPU can ask the NIC how many nanoseconds ago that was

The CPU can indeed ask the NIC what time it is, but then the CPU has to estimate how long ago the NIC answered the question. If the PCI bus is in L1, it will take 10s to 100s of microseconds (no hard upper bound; could be forever) to train up to L0. The host has to determine this delay and compensate for it, because PCI bus transition is much longer than the desired error in PTP. The easiest way is to repeatedly read the time, discard the outliers, and divide the estimated delay in half. This technique is used by various realtime ethernet stacks. You will note that this is effectively the same as disabling ASPM. This is also why they invented PCIe 3.0 PTM.

Re: Clock synchronization is a nightmare

#144

Post by layer8 »

> The good news is that the International Bureau of Weights and Measures has decided to stop adding leap seconds by 2035.

This is not entirely correct. What has been agreed is to allow deviations of more than one second after 2035, so that clocks have to be adjusted less frequently (on the order of every 50-100 years is the intention). However, the allowable deviation, and how to adjust clocks when it is exceeded, has yet to be decided.

Re: Clock synchronization is a nightmare

#145

Post by danzheng »

koudelka wrote: post #2

the Huygens algorithm is also worth a look https://www.usenix.org/system/files/conference/nsdi18/nsdi18...

A very clever part of the HUYGENS algorithm is that it doesn’t just sync clocks pair-wise, it leverages a natural network effect where a group of pair-wise synchronized clocks becomes transitively synchronized, helping reduce errors further without requiring specialized hardware. That’s one of the key reasons it can achieve ~100 nanoseconds of software-based sync on commodity networks.

The authors’ work forms the basis of what the team at Clockwork.io is building, enabling accurate one-way delay measurements (rather than just RTT/2) that improve latency visibility and telemetry across CPU and GPU infrastructure

Re: Clock synchronization is a nightmare

#146

Post by danzheng »

One thing missing in the blogpost is in practice you see many large orgs, especially in finance, living with multiple time domains. For example, on-prem trading systems almost always use PTP or PPS for sub-microsecond timestamping, often on dedicated networks to reduce jitter (for meeting regulatory requirements like MiFID II and CAT) while the rest of their infra (in on-prem and cloud) just runs NTP for millisecond-class sync. Both protocols are fundamentally sensitive to network conditions — the mean offset may look fine, but outliers due to congestion/jitter can be very poor.

The consequence of having multiple time domains is pretty painful when you need to reconcile logs or transaction histories across systems with different sync accuracy. Millisecond NTP logs and sub-microsecond PTP logs don’t line up cleanly, so correlating events end-to-end can become guesswork rather than deterministic ordering.

If you want reliable cross-system telemetry and audit trails, you'll need a single, high-accuracy time sync approach across your whole stack.

Re: Clock synchronization is a nightmare

#147

Post by NelsonMinar »

jasonwatkinspdx wrote: post #51
NelsonMinar wrote: post #48

On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at ho…

At this point the only clock in my life that doesn't auto set is the one on my stove, and that's because I abhor internet connected kitchen appliances.

One of the best features of my microwave is the ability to turn the clock off entirely. If it doesn't set itself I'd rather just not see it!

Re: Clock synchronization is a nightmare

#148

Post by ssl-3 »

otterley wrote: post #124
ssl-3 wrote: post #95

Earlier quoted context omitted.

Both of the WWVB clocks I've owned have been very fickle about how they're placed because RF be that way sometimes, and Colorado isn't exactly nearby to my location in Ohio. The first manufactured GPS clock I owned (as in: switch it on and time is shown on a dedicated display) was in a 2007 Honda. But a firmware bug ruined that clock: https://didhondafixtheclocks.com/ And even after it began displaying the right time…

The WWVB clocks are around the AM band, which means they carry a great distance despite their lower transmission power, but only at nighttime. Ohio is nothing; the signal needs to make it to the southern reaches of Florida.

Yeah, I "know" how it is supposed to work. I "knew" that back then when I bought my first WWVB-receiving clock, too.

And the placement was still fickle.

It's simple to observe:

A) Purchase and install clock. Wait (days). Observe failure to chooch.

B) Move clock. Wait (hours). Observe correct operation.

My world is ultimately bounded by reality, not theory: When it works in one position but not another, then that's the only reality I have to work with.

Re: Clock synchronization is a nightmare

#149

Post by Dylan16807 »

jeffbee wrote: post #143
Dylan16807 wrote: post #113

Earlier quoted context omitted.

> The control loop for host PTP benefits from regularity. How much regularity? If you sent PTP packets with 5 milliseconds of randomness in the scheduling, does that cause real problems? It's still going to have an accurate timestamp. > instruct the host operating system to disable high-exit-latency sleep features Why, though? You didn't explain this. As long as the packet got timestamped when it arrived, the CPU can…

> PTP packets with 5 milliseconds of randomness in the scheduling This should not matter, unless you are a 5G telecom operator running at a high frequency. Gaussian noise in the master is not important to PTP. Being a master is easier than being a slave. If you are running PTP at 128 Hz like a telecom, delays that large might lead to slaves resetting their state machines, which would blow the whole thing up. > The CP…

> You will note that this is effectively the same as disabling ASPM

You need to wake everything up for certain measurements, but you don't need to disable power saving wholesale.

Re: Clock synchronization is a nightmare

#150

Post by [deleted] »

willis936 wrote: post #93
RossBencina wrote: post #89

Earlier quoted context omitted.

Out of interest, how do you measure a sub-10ps phase lock between devices 50km apart?

The standards-compliant endpoints do all of the work. They count clock cycles for ping pong messages and share with each other the length of time so time-of-flight is tracked and compensated for.

[deleted]
Post reply on HN