Live data from Hacker News

Clock synchronization is a nightmare

arpitbhayani.me

81–90 of 165 posts

Re: Clock synchronization is a nightmare

#81

On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at ho…

At this point the only clock in my life that doesn't auto set is the one on my stove, and that's because I abhor internet connected kitchen appliances.

Good ol' Oven Standard Time (OST).

Re: Clock synchronization is a nightmare

#82

Timesync isn’t a nightmare at all. But it is a deep rabbit hole. The best approach, imho, is to abandon the concept of a global time. All timestamps are wrt a specific clock. That clock will skew at a rate that varies with time. You can, hopefully, rely on any particular clock being monotonous! My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any cl…

"I kinda don’t like PTP. Too complicated and requires specialized hardware."

?????

I run PTP on everything from RPI's to you name it, over fiber, ethernet, etc.

The main thing hardware gives is filtration of PTP packets or hardware timestamping.

Neither is actually required, though some software has decided to require it.

Additionally, something like 99% of sold gigabit or better chipsets since 2012 support it (I210 et al)

Re: Clock synchronization is a nightmare

#85
post #3

PTP requires support not only on your network, but also on your peripheral bus and inside your CPU. It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states.

PTP does not require support on your network beyond standard ethernet packet forwarding when used in ethernet mode.

In multicast IP mode, with multiple switches, it requires what anything running multicast between switches/etc would require (IE some form of IGMP snopping or multicast routing or .....)

In unicast IP mode, it requires nothing from your network.

Therefore, i have no idea what it means to "require support on the network".

I have used both ethernet and multicast PTP across a complete mishmash of brands and types and medias of switches, computers, etc, with no issues.

The only thing that "support" might improve is more accurate path delay data through transparent clocks. If both master and slave do accurate hardware timestamping already, and the path between them is constant, it is easily possible to get +-50 nanoseconds without any transparent clock support.

Here is the stats from a random embedded device running PTP i just accessed a second ago:

  Reference ID    : 50545030 (PTP0)
  Stratum         : 1
  Ref time (UTC)  : Sun Dec 28 02:47:25 2025
  System time     : 0.000000029 seconds slow of NTP time
  Last offset     : -0.000000042 seconds
  RMS offset      : 0.000000034 seconds
  Frequency       : 8.110 ppm slow
  Residual freq   : -0.000 ppm
  Skew            : 0.003 ppm
So this embedded ARM device, which is not special in any way, is maintaining time +-35ns of the grandmaster, and currently 30ns of GPS time.

The card does not have an embedded hardware PTP clock, but it does do hardware timestamp and filtering.

This grandmaster is an RPI with an intel chipset on it and the PPS input pin being used to discipline the chipset's clock. It stays within +-2ns (usually +-1ns) of GPS time.

Obviously, holdover sucks, but not the point :)

This qualifies as better-than-NTP for sure, and this setup has no network support. No transparent clocks, etc. These machines have multiple media transitions involved (fiber->ethernet), etc.

The main thing transparent clock support provides in practice is dealing with highly variable delay. Either from mode of transport, number of packet processors in between your nodes, etc. Something that causes the delay to be hard to account for.

The ethernet packet processing in ethernet mode is being handled in hardware by the switches and basically all network cards. IP variants would probably be hardware assisted but not fully offloaded on all cards, and just ignored on switches (assuming they are not really routers in disguise).

The hardware timestamping is being done in the card (and the vast majority of ethernet cards have supported PTP harware timestamping for >1 decade at this point), and works perfectly fine with deep CPU sleep states.

Some don't do hardware filtering so they essentially are processing more packets that necessary but .....

Re: Clock synchronization is a nightmare

#86
post #73

I highly recommend anyone to look up how PTP works and how it compares to NTP. Clock sync is very interesting. When I joined an HFT company, first thing I did was understand this stuff. We care about it a lot[1]. If you want a specific question to answer, answer this: why does PTP need hardware timestamping to achieve high precision (where the network card itself assigns timestamps to packets, rather than having the…

> So do a lot of people. For example audio engineers.

Indeed. PTP (various, not-necessarily compatible, versions) is at the core of modern ethernet-based audio networking: Dante (proprietary, PTP: IEEE 1588 v1), AVB (IEEE standard, PTP: 802.1AS), AES67 (AES standard, PTP: IEEE 1588 v2). And now the scope of the AVB protocol stack has been expanded to TSN for industrial and automotive time sensitive network applications.

Re: Clock synchronization is a nightmare

#87
post #52

Earlier quoted context omitted.

You can if you just run PTP (almost) entirely on your NIC. The best PTP implementations take their packet timestamps at the MAC on the NIC and keep time based on that. Nothing about CPU processing is time-critical in that case.

Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. The control loop for host PTP benefits from regularity. Anyway NICs that support PTP hardware timestamping may also use PCI LTR (latency tolerance reporting) to instruct the host operating system to disable high-exit-latency sleep features, and popular operating systems respect that.

"Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. "

I'm sorry, this is just moving the goalposts.

You said "It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states."

This is flat wrong, as pointed out.

Now you are pedantically arguing that some NIC's that do PTP hardware timestamping might also use a feature that some operating systems might respect.

That's a very far cry from "It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states".

In most cases, people would just say "hey i was wrong about that but there are cases that i think matter where it falls down".

Re: Clock synchronization is a nightmare

#88

Earlier quoted context omitted.

Of course, you can do this in good conditions. The extremely powerful part that TrueTime brings is how the system degrades when something goes wrong. If everyone is synced to +/- 20ns, that's great. Then when someone flies over your datacenter with an GPS jammer (purposeful or accidental), this needs to not be a bad day where suddenly database transactions happen out of order, or you have an outage. The other benefit…

> Then when someone flies over your datacenter with an GPS jammer (purposeful or accidental), this needs to not be a bad day where suddenly database transactions happen out of order, or you have an outage. Most NTP/PTP appliances have internal clocks that are OCXO or rubidium that have holdover (even for several days). If time is that important to you then you'll have them, plus perhaps some fibre connections to othe…

> fibre connections to other sites that are hopefully out of range of the jamming.

I guess it's not inconceivable that eventually there's a global clock network using a White-Rabbit-like protocol over dedicated fibre. But if you have to worry about GPS jamming you probably have to worry about undersea cable cutting too.

Re: Clock synchronization is a nightmare

#89
post #65
post #9

Another protocol that's not mentioned is PPS and its variants, such as WhiteRabbit. A regular pulse is emitted from a specialized high-precision device, possibly over a specialized high-precision network. Enables picosecond accuracy (or at least sub-nano).

As a user of WhiteRabbit, I can confirm a sub-10ps sync (two clocks phase lock) over 50km fiber connection for variable temperature of fiber (biggest problem of clock sync over fibers is temperature induced length change of the fiber itself, which needs to be measured and compensated).

Out of interest, how do you measure a sub-10ps phase lock between devices 50km apart?

Re: Clock synchronization is a nightmare

#90
post #37

Clock sync is such a nightmare in robotics. Most OSes happily will skew/jump to get the time correct. Time jumps (especially backwards) will crash most robotics stacks. You might decide to ensure that you have synced time before starting the stack. Great, now your timestamps are mostly accurate, except what happens when you've used GPS as your time source, and you start indoors? Robot hangs forever. Hot take: I've se…

C++11 distinguishes system_clock from steady_clock. As you say, using system_clock is a bug.
Post reply on HN