Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clock synchronization is a nightmare

arpitbhayani.me

Post reply on HN
Print view View original

61–70 of 165 posts

Re: Clock synchronization is a nightmare

#61

Post by a_t48 »

michaelt wrote: post #44
a_t48 wrote: post #37

Clock sync is such a nightmare in robotics. Most OSes happily will skew/jump to get the time correct. Time jumps (especially backwards) will crash most robotics stacks. You might decide to ensure that you have synced time before starting the stack. Great, now your timestamps are mostly accurate, except what happens when you've used GPS as your time source, and you start indoors? Robot hangs forever. Hot take: I've se…

If your requirements are “must have accurate time, must start with an inaccurate time, must not step time during operation, no atomic clocks, must not require a network connection, or a WWVB signal, must work without a GPS signal” then yes, you need to relax your requirements. But it doesn’t have to be the first requirement you relax.

If it has a GPS already, it’s really easy to fall into the trap of just using it, but point taken. Then main requirement is accurate moment to moment time. Using GPS as the master clock mostly makes sense there.

Re: Clock synchronization is a nightmare

#62

Post by idorosen »

vlovich123 wrote: post #39
idorosen wrote: post #27

Earlier quoted context omitted.

Alternatively, you could guarantee the same synchronization using PPS and PTP to each host's DCD pin of their serial port or to specialized hardware such as modern PTP-enabled smart NICs/FPGAs that can accept PPS input. GPS+PPS gets you to within 20-80ns global synchronization depending on implementation (assuming you're all mostly in the same inertial frame), and allows you to make much stronger guarantees than True…

> and allows you to make much stronger guarantees than TrueTime (due to higher precision distributed ordering guarantees, which translate to lower latency and higher throughput distributed writes). TrueTime is the software algorithm for managing the timestamps. It’s agnostic to the accuracy of the underlying time source. If it was inaccurate then you get looser bounds and as you note higher latency. Google already do…

Yup! I was referring to the original TrueTime/Spanner papers, not whatever's currently deployed. The original paper makes reference to distributed ordering guarantees at the milliseconds' scale precision, which implies many more transactions in flight in the uncertain state and coarser distributed ordering guarantees than the much tighter upper bound you can set with nanoseconds' precision and microseconds' comms latency...

Re: Clock synchronization is a nightmare

#63

Post by senfiaj »

maximinus_thrax wrote: post #8

I wouldn't say it's a 'nightmare'. It's just more complicated than what regular folk think computers work when it comes to time sync. There's nothing nightmareish or scary about this, it's just using the best solution for your scenario, understanding limitations and adjusting expectations/requirements accordingly, perhaps relaxing consistency requirements. I worked on the NTP infra for a very large organization some…

Maybe. But I remember one game developer told that they face even a more challenging problem, which is the synchronization between players in multiplayer real-time games. Just imagine different users having significantly different network latencies in a multiplayer shooter where a couple milliseconds can be decisive. Someone makes a headshot when the game state is already outdated. If you think about this you can appreciate how it's complicated just to make the gameplay at least not awful...

Re: Clock synchronization is a nightmare

#64

Post by nuccy »

shomp wrote: post #50

Absolute synchronization impossible?? Challenge accepted.

Nature (laws of physics) is agains you on this: it is in fact impossible for everyone. What is in sync for some observers can be out of sync for others (depends on where they are, i.e. gravity, and how they relatively move). See general and special relativity principle of simultaneity [1].

1. https://en.wikipedia.org/wiki/Relativity_of_simultaneity

Re: Clock synchronization is a nightmare

#65

Post by nuccy »

mgaunard wrote: post #9

Another protocol that's not mentioned is PPS and its variants, such as WhiteRabbit. A regular pulse is emitted from a specialized high-precision device, possibly over a specialized high-precision network. Enables picosecond accuracy (or at least sub-nano).

As a user of WhiteRabbit, I can confirm a sub-10ps sync (two clocks phase lock) over 50km fiber connection for variable temperature of fiber (biggest problem of clock sync over fibers is temperature induced length change of the fiber itself, which needs to be measured and compensated).

Re: Clock synchronization is a nightmare

#66

Post by idorosen »

structural wrote: post #59
idorosen wrote: post #27

Earlier quoted context omitted.

Alternatively, you could guarantee the same synchronization using PPS and PTP to each host's DCD pin of their serial port or to specialized hardware such as modern PTP-enabled smart NICs/FPGAs that can accept PPS input. GPS+PPS gets you to within 20-80ns global synchronization depending on implementation (assuming you're all mostly in the same inertial frame), and allows you to make much stronger guarantees than True…

Of course, you can do this in good conditions. The extremely powerful part that TrueTime brings is how the system degrades when something goes wrong. If everyone is synced to +/- 20ns, that's great. Then when someone flies over your datacenter with an GPS jammer (purposeful or accidental), this needs to not be a bad day where suddenly database transactions happen out of order, or you have an outage. The other benefit…

In summary, with different business requirements you would build a different technical solution.

Re: Clock synchronization is a nightmare

#67

Post by johnisgood »

josephg wrote: post #53

> When two transactions happen at nearly the same time on different nodes, the database must determine which happened first. If clocks are out of sync, the database might order them incorrectly, violating consistency guarantees. This is only true if you use wall clock time as part of your database’s consistency algorithm. Generally I think this is a huge mistake. It’s almost always much easier to swap to a logical cl…

Related in many ways: https://www.erlang.org/docs/22/apps/erts/time_correction

Also I still remember having fun with the "Determine the order of events by saving a tuple containing monotonic time and a strictly monotonically increasing integer as follows" part.

Re: Clock synchronization is a nightmare

#68

Post by jasonwatkinspdx »

jaggederest wrote: post #60
amelius wrote: post #57

Earlier quoted context omitted.

Same here. I wish there was an easy way around it (that doesn't require me to play sysadmin in my spare time).

In the 80s my uncle had digital clocks that used an antenna to tune into the atomic clock time signal that (was/is?) broadcast nationwide. I've long wished that it was incorporated into stoves, microwaves, essentially everything that isn't an internet device (yet... sigh) Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of…

> Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of a pin these days.

Unfortunately there's no real way to cheat physics as far as shrinking a wavelength goes. With RF antennas about the best you can do is a major dimension 1/10th the frequency of interest.

Re: Clock synchronization is a nightmare

#69

Post by kccqzy »

vlovich123 wrote: post #38
kccqzy wrote: post #35

Earlier quoted context omitted.

That’s exactly what I’m saying but you simply provided more details. TrueTime guarantees clocks are well synchronized: and of course that means synchronized to a reasonable upper bound. It’s no more possible for clocks to be absolutely synchronized, than for two line segments drawn independently to have absolutely the same length.

> you can compare them to determine exactly which commit happened first This is the part I was referring to. You cannot just compare timestamps and know which happened first. You have to actually handle the case where you don’t know if there’s a happens before relationship between the timestamps. Thats a very important distinction

I quote from Spanner docs at https://docs.cloud.google.com/spanner/docs/true-time-externa...

> External consistency states that Spanner executes transactions in a manner that is indistinguishable from a system in which the transactions are executed serially, and furthermore, that the serial order is consistent with the order in which transactions can be observed to commit. Because the timestamps generated for transactions correspond to the serial order, if any client sees a transaction T2 start to commit after another transaction T1 finishes, the system will assign a timestamp to T2 that is higher than T1's timestamp.

Of course there is always the edge case where two commits have the same commit timestamp. Therefore from the perspective of Spanner, they happen simultaneously and there is no way to determine which happens first. But there is no need to. There is no causality relationship between them. If you insist, you can arbitrarily assign a happens-before relationship in your own code and nothing will break.

Re: Clock synchronization is a nightmare

#70

Post by raron »

NelsonMinar wrote: post #48

On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at ho…

> clock sync for civilians has never been easier

I don't think civilian clock synchronization was an issue since a long time ago.

DCF77 and WWVB has been around for more than 50 years. You could use some cheap electronics and get well below millisecond accuracy. GPS has been fully operational for 30 years, but it needs more expensive device.

I suspect you could even get below 1 sec accuracy using a watch with a hacking movement and listening to radio broadcast of time beeps / pips.

Post reply on HN