Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clock synchronization is a nightmare

arpitbhayani.me

Post reply on HN
Print view View original

91–100 of 165 posts

Re: Clock synchronization is a nightmare

#91

Post by baby_souffle »

dmazin wrote: post #73

I highly recommend anyone to look up how PTP works and how it compares to NTP. Clock sync is very interesting. When I joined an HFT company, first thing I did was understand this stuff. We care about it a lot[1]. If you want a specific question to answer, answer this: why does PTP need hardware timestamping to achieve high precision (where the network card itself assigns timestamps to packets, rather than having the…

Not just audio, anybody in the live events / production space needs all equipment marching in lock step.

Re: Clock synchronization is a nightmare

#92

Post by RossBencina »

forrestthewoods wrote: post #29

Timesync isn’t a nightmare at all. But it is a deep rabbit hole. The best approach, imho, is to abandon the concept of a global time. All timestamps are wrt a specific clock. That clock will skew at a rate that varies with time. You can, hopefully, rely on any particular clock being monotonous! My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any cl…

> I kinda don’t like PTP. Too complicated and requires specialized hardware.

In my view the specialised hardware is just a way to get more accurate transmission and arrival timestamps. That's useful whether or not you use PTP.

> My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any clock to any clock. This is a lossy conversion that has jitter and can change with time.

This sounds like the "peer to peer" equivalent to PTP. It would require every node to maintain state about it's estimate (skew, slew, variance) of every other clock. I like the concept, but obviously it adds complexity to end-stations beyond what PTP requires (i.e. increases the hardware cost of embedded implementations). Such a system would also need to model the network topology, or control routing (as PTP does), because packets traversing different routes to the same host will experience different delay and jitter statistics.

> TicSync is cool

I hadn't seen this before, but I have implemented similar convex-hull based methods for clock recovery. I agree this is obviously a good approach. Thanks for sharing.

Re: Clock synchronization is a nightmare

#93

Post by willis936 »

RossBencina wrote: post #89
nuccy wrote: post #65

Earlier quoted context omitted.

As a user of WhiteRabbit, I can confirm a sub-10ps sync (two clocks phase lock) over 50km fiber connection for variable temperature of fiber (biggest problem of clock sync over fibers is temperature induced length change of the fiber itself, which needs to be measured and compensated).

Out of interest, how do you measure a sub-10ps phase lock between devices 50km apart?

The standards-compliant endpoints do all of the work. They count clock cycles for ping pong messages and share with each other the length of time so time-of-flight is tracked and compensated for.

Re: Clock synchronization is a nightmare

#94

Post by forrestthewoods »

RossBencina wrote: post #92
forrestthewoods wrote: post #29

Timesync isn’t a nightmare at all. But it is a deep rabbit hole. The best approach, imho, is to abandon the concept of a global time. All timestamps are wrt a specific clock. That clock will skew at a rate that varies with time. You can, hopefully, rely on any particular clock being monotonous! My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any cl…

> I kinda don’t like PTP. Too complicated and requires specialized hardware. In my view the specialised hardware is just a way to get more accurate transmission and arrival timestamps. That's useful whether or not you use PTP. > My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any clock to any clock. This is a lossy conversion that has jitter and ca…

> This sounds like the "peer to peer" equivalent to PTP. It would require every node to maintain state about it's estimate (skew, slew, variance) of every other clock.

Well, it requires having the conversion function for each edge in the traversed path. And such function needs to exist only at the location(s) performing the conversion.

> obviously it adds complexity to end-stations beyond what PTP requires

If you have PTP and it works then stick with it. If you’re trying to timesync a network of wearable devices then you don’t have PTP stamping hardware.

> because packets traversing different routes

Fair callout. It’s probably a more useful model for less internty use cases. Of which there are many!

For example when trying to timesync a collection of different sensors on different devices/microcontrollers.

Roboticists like CanBus and Ethercat. But even that is kinda overkill imho. TicSync can get you tens of microseconds of precision in user space.

Re: Clock synchronization is a nightmare

#95

Post by ssl-3 »

raron wrote: post #70
NelsonMinar wrote: post #48

On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at ho…

> clock sync for civilians has never been easier I don't think civilian clock synchronization was an issue since a long time ago. DCF77 and WWVB has been around for more than 50 years. You could use some cheap electronics and get well below millisecond accuracy. GPS has been fully operational for 30 years, but it needs more expensive device. I suspect you could even get below 1 sec accuracy using a watch with a hacki…

Both of the WWVB clocks I've owned have been very fickle about how they're placed because RF be that way sometimes, and Colorado isn't exactly nearby to my location in Ohio.

The first manufactured GPS clock I owned (as in: switch it on and time is shown on a dedicated display) was in a 2007 Honda.

But a firmware bug ruined that clock: https://didhondafixtheclocks.com/

And even after it began displaying the right time again, it had the wrong date. It was offset by years and years, which was OK-ish, but also by several months.

Having the date offset by months caused the HVAC to behave in strange incurable ways because it expected the sun to be in positions where it was not.

But NTP? NTP has never been fickle for me, even in the intermittently-connected dialup days I experienced ~30 years ago: If I can get to the network occasionally, then I can connect to a few NTP servers and keep a local clock reasonably-accurate.

NTP has been resolutely awesome for me.

Re: Clock synchronization is a nightmare

#96

Post by dmazin »

RossBencina wrote: post #86
dmazin wrote: post #73

I highly recommend anyone to look up how PTP works and how it compares to NTP. Clock sync is very interesting. When I joined an HFT company, first thing I did was understand this stuff. We care about it a lot[1]. If you want a specific question to answer, answer this: why does PTP need hardware timestamping to achieve high precision (where the network card itself assigns timestamps to packets, rather than having the…

> So do a lot of people. For example audio engineers. Indeed. PTP (various, not-necessarily compatible, versions) is at the core of modern ethernet-based audio networking: Dante (proprietary, PTP: IEEE 1588 v1), AVB (IEEE standard, PTP: 802.1AS), AES67 (AES standard, PTP: IEEE 1588 v2). And now the scope of the AVB protocol stack has been expanded to TSN for industrial and automotive time sensitive network applicatio…

Yeah, the audio engineer then talked my ear off about networking!

Re: Clock synchronization is a nightmare

#97

Post by winrid »

Uehreka wrote: post #77
NelsonMinar wrote: post #48

On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at ho…

I was doing something at work that involved calculating round trip times from/to Android devices, and learned that although it should be possible for NTP to sync clocks with below-second precision, in practice many of the Android devices I was working with (mostly Pixels 2-7) were off from my server and each other by up to 5 seconds, which blew my mind.

Yeah it can only do so much with varying latency

Re: Clock synchronization is a nightmare

#98

Post by toast0 »

Uehreka wrote: post #77
NelsonMinar wrote: post #48

On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at ho…

I was doing something at work that involved calculating round trip times from/to Android devices, and learned that although it should be possible for NTP to sync clocks with below-second precision, in practice many of the Android devices I was working with (mostly Pixels 2-7) were off from my server and each other by up to 5 seconds, which blew my mind.

It's hard to keep a phone's clock closely synchronized because they experience a lot of temperature swings, going between pockets and hands and open air and sometimes in direct sun, and the processor goes between idle and 100% as well.

Once you get to internationa phones, you'll have places where the phone does not include all timezones and specifically is missing the actual local timezone, so automatic sync is typically disabled so that the time can be set so that the displayed time matches local time... even if that means the system time is not correct.

Re: Clock synchronization is a nightmare

#99

Post by namibj »

jasonwatkinspdx wrote: post #68
jaggederest wrote: post #60

Earlier quoted context omitted.

In the 80s my uncle had digital clocks that used an antenna to tune into the atomic clock time signal that (was/is?) broadcast nationwide. I've long wished that it was incorporated into stoves, microwaves, essentially everything that isn't an internet device (yet... sigh) Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of…

> Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of a pin these days. Unfortunately there's no real way to cheat physics as far as shrinking a wavelength goes. With RF antennas about the best you can do is a major dimension 1/10th the frequency of interest.

There are many DCF77 receivers in Germany that are contained in a square box that's barely large enough for a AA battery; the rest of the square contains the motor/gears and the electronics/receiver (incl. a ferrite loopstick antenna).

The wavelength is around 3.8km...

Re: Clock synchronization is a nightmare

#100

Post by jeffbee »

DannyBee wrote: post #87
jeffbee wrote: post #52

Earlier quoted context omitted.

Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. The control loop for host PTP benefits from regularity. Anyway NICs that support PTP hardware timestamping may also use PCI LTR (latency tolerance reporting) to instruct the host operating system to disable high-exit-latency sleep features, and popular operating systems respect that.

"Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. " I'm sorry, this is just moving the goalposts. You said "It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states." This is flat wrong, as pointed out. Now you are pedantically arguing that some NIC's that do PTP hardware timestamping might also use a feat…

I see nothing in your pair of unnecessarily belligerent comments that actually contradicts what I said. There are host-side features that enable the clock discipline you are observing, even if you are apparently not aware of them.
Post reply on HN