Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clock synchronization is a nightmare

arpitbhayani.me

Post reply on HN
Print view View original

101–110 of 165 posts

Re: Clock synchronization is a nightmare

#101

Post by namibj »

RossBencina wrote: post #88
throw0101c wrote: post #75

Earlier quoted context omitted.

> Then when someone flies over your datacenter with an GPS jammer (purposeful or accidental), this needs to not be a bad day where suddenly database transactions happen out of order, or you have an outage. Most NTP/PTP appliances have internal clocks that are OCXO or rubidium that have holdover (even for several days). If time is that important to you then you'll have them, plus perhaps some fibre connections to othe…

> fibre connections to other sites that are hopefully out of range of the jamming. I guess it's not inconceivable that eventually there's a global clock network using a White-Rabbit-like protocol over dedicated fibre. But if you have to worry about GPS jamming you probably have to worry about undersea cable cutting too.

Good thing cesium fountains are very accurate then...

Re: Clock synchronization is a nightmare

#102

Post by namibj »

RossBencina wrote: post #89
nuccy wrote: post #65

Earlier quoted context omitted.

As a user of WhiteRabbit, I can confirm a sub-10ps sync (two clocks phase lock) over 50km fiber connection for variable temperature of fiber (biggest problem of clock sync over fibers is temperature induced length change of the fiber itself, which needs to be measured and compensated).

Out of interest, how do you measure a sub-10ps phase lock between devices 50km apart?

Run 2 or 3 separate concurrent sync's and statistically compare the resulting clocks, for example.

Re: Clock synchronization is a nightmare

#103

Post by vel0city »

toast0 wrote: post #98
Uehreka wrote: post #77

Earlier quoted context omitted.

I was doing something at work that involved calculating round trip times from/to Android devices, and learned that although it should be possible for NTP to sync clocks with below-second precision, in practice many of the Android devices I was working with (mostly Pixels 2-7) were off from my server and each other by up to 5 seconds, which blew my mind.

It's hard to keep a phone's clock closely synchronized because they experience a lot of temperature swings, going between pockets and hands and open air and sometimes in direct sun, and the processor goes between idle and 100% as well. Once you get to internationa phones, you'll have places where the phone does not include all timezones and specifically is missing the actual local timezone, so automatic sync is typic…

Don't a lot of cellular networks rely on highly synchronized clocks to properly handle TDMA-style transmissions? Shouldn't they be very in sync with the towers' times?

Re: Clock synchronization is a nightmare

#104

Post by maximinus_thrax »

senfiaj wrote: post #63
maximinus_thrax wrote: post #8

I wouldn't say it's a 'nightmare'. It's just more complicated than what regular folk think computers work when it comes to time sync. There's nothing nightmareish or scary about this, it's just using the best solution for your scenario, understanding limitations and adjusting expectations/requirements accordingly, perhaps relaxing consistency requirements. I worked on the NTP infra for a very large organization some…

Maybe. But I remember one game developer told that they face even a more challenging problem, which is the synchronization between players in multiplayer real-time games. Just imagine different users having significantly different network latencies in a multiplayer shooter where a couple milliseconds can be decisive. Someone makes a headshot when the game state is already outdated. If you think about this you can app…

And yet it has been done with different approaches, it's complicated, yes, as I said. But nothing nightmarish about it.

Re: Clock synchronization is a nightmare

#105

Post by forrestthewoods »

DannyBee wrote: post #82
forrestthewoods wrote: post #29

Timesync isn’t a nightmare at all. But it is a deep rabbit hole. The best approach, imho, is to abandon the concept of a global time. All timestamps are wrt a specific clock. That clock will skew at a rate that varies with time. You can, hopefully, rely on any particular clock being monotonous! My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any cl…

"I kinda don’t like PTP. Too complicated and requires specialized hardware." ????? I run PTP on everything from RPI's to you name it, over fiber, ethernet, etc. The main thing hardware gives is filtration of PTP packets or hardware timestamping. Neither is actually required, though some software has decided to require it. Additionally, something like 99% of sold gigabit or better chipsets since 2012 support it (I210…

Robots and VR headsets and wearables and microcontrollers and sensors and trackers and Linux and Windows oh my!

Re: Clock synchronization is a nightmare

#106

Post by tobias2014 »

Enginerrrd wrote: post #72
nuccy wrote: post #64

Earlier quoted context omitted.

Nature (laws of physics) is agains you on this: it is in fact impossible for everyone. What is in sync for some observers can be out of sync for others (depends on where they are, i.e. gravity, and how they relatively move). See general and special relativity principle of simultaneity [1]. 1. https://en.wikipedia.org/wiki/Relativity_of_simultaneity

I think you just nerd-sniped me but I’m not convinced it’s impossible to assign a consistent ordering to events with relativistic separations. For starters, the spacetime interval between two events IS a Lorentz invariant quantity. That could probably be used to establish a universal order for timelike separations between events. I suspect that you could use a reference clock, like a pulsar or something to act as an…

For any space-like event you can find reference frames where things happen in different order. For the time-like situation you described the order indeed exists within the cone, which is to say that causality exists.

Re: Clock synchronization is a nightmare

#107

Post by tombert »

dmazin wrote: post #73

I highly recommend anyone to look up how PTP works and how it compares to NTP. Clock sync is very interesting. When I joined an HFT company, first thing I did was understand this stuff. We care about it a lot[1]. If you want a specific question to answer, answer this: why does PTP need hardware timestamping to achieve high precision (where the network card itself assigns timestamps to packets, rather than having the…

I find time accuracy to be ridiculously interesting, and I have had to talk myself out of buying those a used atomic clock to play with [1]. I think precision time is very cool, and a small part of me wants to create the most overly engineered wall-clock using a Raspberry Pi or something to have sub-microsecond level accuracy.

Sadly, they're generally just a bit too expensive for me to justify it as a toy.

I don't work in trading (though not for lack of trying on my end), so most of the stuff I work on has been a lot more about "logical clocks", which are cool in their own right, but I have always wondered how much more efficient we could be if we had nanosecond-level precision to guarantee that locks are almost always uncontested.

[1] I'm not talking about those clocks that radio to Colorado or Greenwich, I mean the relatively small ones that you can buy that run locally.

Re: Clock synchronization is a nightmare

#108

Post by Dylan16807 »

ssl-3 wrote: post #95
raron wrote: post #70

Earlier quoted context omitted.

> clock sync for civilians has never been easier I don't think civilian clock synchronization was an issue since a long time ago. DCF77 and WWVB has been around for more than 50 years. You could use some cheap electronics and get well below millisecond accuracy. GPS has been fully operational for 30 years, but it needs more expensive device. I suspect you could even get below 1 sec accuracy using a watch with a hacki…

Both of the WWVB clocks I've owned have been very fickle about how they're placed because RF be that way sometimes, and Colorado isn't exactly nearby to my location in Ohio. The first manufactured GPS clock I owned (as in: switch it on and time is shown on a dedicated display) was in a 2007 Honda. But a firmware bug ruined that clock: https://didhondafixtheclocks.com/ And even after it began displaying the right time…

On the one hand, some sloppy GPS units fail on a 20 year schedule. On the other hand, a bunch of things using NTP are going to fail in about ten years. (2036 rather than 2038 because reasons)

Re: Clock synchronization is a nightmare

#110

Post by sreekanth850 »

In physics, time is local and relative, independent events don’t need a global ordering. Distributed databases shouldn’t require one either. The idea of a single global time comes from 1980s single-node database semantics, where serializability implied one universal execution order. When that model was lifted into distributed systems, researchers introduced global clocks and timestamp coordination to preserve those guarantees, not because distributed systems fundamentally need it. It’s time we rethink this., Only operations that touch the same piece of data require ordering. Everything else should follow causality like the physical universe, independent events don’t need to agree on sequence, only dependent ones do. Global clocks exist because some databases forced serializable cross-object transactions onto distributed systems, not because nature requires it. Edit: I welcome for a discussion with people who disagree and downvote.
Post reply on HN