Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clock synchronization is a nightmare

arpitbhayani.me

Post reply on HN
Print view View original

31–40 of 165 posts

Re: Clock synchronization is a nightmare

#31

Post by lll-o-lll »

georgelyon wrote: post #12

Unfortunate that the author doesn’t bring up FoundationDB version stamps, which to me feel like the right solution to the problem. Essentially, you can write a value you can’t read until after the transaction is committed and the synchronization infrastructure guarantees that value ends up being monotonically increasing per transaction. They use similar “write only” operations for atomic operations like increment.

Yes. A consistent total ordering is what you need (want) in distributed computing. Ultimately, causality is what is important, but consistent ordering of concurrent operations makes things much easier to work with.

Re: Clock synchronization is a nightmare

#32

Post by AlotOfReading »

idorosen wrote: post #27
kccqzy wrote: post #25

Earlier quoted context omitted.

If you are an engineer at Google dealing with Spanner, then you can in fact assume clocks are well synchronized and can use timestamps for synchronization. If you get commit timestamps from Spanner you can compare them to determine exactly which commit happened first. That’s a stronger guarantee than the typical Serializable database like postgresql: https://www.postgresql.org/docs/current/transaction-iso.html... Tha…

Alternatively, you could guarantee the same synchronization using PPS and PTP to each host's DCD pin of their serial port or to specialized hardware such as modern PTP-enabled smart NICs/FPGAs that can accept PPS input. GPS+PPS gets you to within 20-80ns global synchronization depending on implementation (assuming you're all mostly in the same inertial frame), and allows you to make much stronger guarantees than True…

Truetime is based on GPS and local atomic clocks. Google's latest timemasters are even better, around 10ns average.

Re: Clock synchronization is a nightmare

#33

Post by libraryofbabel »

hinkley wrote: post #24
maximinus_thrax wrote: post #8

I wouldn't say it's a 'nightmare'. It's just more complicated than what regular folk think computers work when it comes to time sync. There's nothing nightmareish or scary about this, it's just using the best solution for your scenario, understanding limitations and adjusting expectations/requirements accordingly, perhaps relaxing consistency requirements. I worked on the NTP infra for a very large organization some…

I took to distributed systems like a duck to water. It was only much later that I figured out that while there are things I can figure out in one minute that took other people five, there were a lot of others that you will have to walk them through step by step or they would never get there. That really explained some interactions I’d had when I was younger. In particular I don’t think the intuitions necessary to do…

> I don’t think the intuitions necessary to do distributed computing well would come to someone who snoozed through physics

Yeah. I was a physics major and it really helped to have had my naive assumptions about time and clocks completely demolished early on by taking classes in special and general relatively. When I eventually found my way into tech a lot of distributed systems concepts that are difficult to other people (clock sync, indeterminate ordering of events, consensus) came quite naturally because of all that early training.

I think it's no accident that distributed systems theory guru Leslie Lamport had written an unpublished book on General Relativity before he wrote the famous Time, Clocks and the Ordering of Events in a Distributed System paper and the Paxos paper. In the former in particular the analogy to special relatively is quite plain to see.

Re: Clock synchronization is a nightmare

#34

Post by ianburrell »

mrkeen wrote: post #17
emptybits wrote: post #15

Earlier quoted context omitted.

Wild. My layperson mind goes to a simple example, which may or may not be possible, but please tell me if this is the gist: Alice and Bob, in different reference frames, both witness events C and D occurring. Alice says C happened before D. Bob says D happened before C. They're both correct. (And good luck synchronizing your watches, Alice and Bob!)

That will be the case when Alice stands close to where C happens, and Bob stands close to where D happens. It's a little trickier to imagine introducing cause-and-effect though. (Alice sees that C caused D to happen, Bob sees that D caused C to happen). I think a "light cone" is the thought-experiment to look up here.

There is distinction between seeing when events happened, and when they really happened. The latter can be reconstructed by an observer.

In special relativity, time is relative and when things actually happened can be different in different frames. Casually linked events are always really in the same order. But disconnected events can be seen in different orders depending on speed of observer.

Re: Clock synchronization is a nightmare

#35

Post by kccqzy »

vlovich123 wrote: post #30
kccqzy wrote: post #25

Earlier quoted context omitted.

If you are an engineer at Google dealing with Spanner, then you can in fact assume clocks are well synchronized and can use timestamps for synchronization. If you get commit timestamps from Spanner you can compare them to determine exactly which commit happened first. That’s a stronger guarantee than the typical Serializable database like postgresql: https://www.postgresql.org/docs/current/transaction-iso.html... Tha…

That’s actually not at all what TrueTime guarantees and assuming they’ve solved a physical impossibility is dangerous technically as a founding assumption for higher level tech (which thankfully Spanner does not do). What TrueTime says is that clocks are synchronized within some delta just like NTP, but that delta is significantly smaller thanks to GPS time sync. That enables applications to have tighter bounds on wa…

That’s exactly what I’m saying but you simply provided more details. TrueTime guarantees clocks are well synchronized: and of course that means synchronized to a reasonable upper bound. It’s no more possible for clocks to be absolutely synchronized, than for two line segments drawn independently to have absolutely the same length.

Re: Clock synchronization is a nightmare

#36

Post by TrainedMonkey »

j_seigh wrote: post #23

Ok,so people use NTP to "synchronize" their clocks and then write applications that assume the clocks are in exact sync and can use timestamps for synchronization, even though NTP can see the clocks aren't always in sync. Do I have that right?

Depending on the application you would generally use PTP to get sub-microsecond accuracy. The real trick is that architecture should tolerate various clocks starting or jumping out of sync and self correct.

Re: Clock synchronization is a nightmare

#37

Post by a_t48 »

Clock sync is such a nightmare in robotics. Most OSes happily will skew/jump to get the time correct. Time jumps (especially backwards) will crash most robotics stacks. You might decide to ensure that you have synced time before starting the stack. Great, now your timestamps are mostly accurate, except what happens when you've used GPS as your time source, and you start indoors? Robot hangs forever.

Hot take: I've seen this and enough other badly configured time sync settings that I want to ban system time from robotics systems - time from startup only! If you want to know what the real world time was for a piece of data after, write what your epoch is once you have a time sync, and add epoch+start time.

Re: Clock synchronization is a nightmare

#38

Post by vlovich123 »

kccqzy wrote: post #35
vlovich123 wrote: post #30

Earlier quoted context omitted.

That’s actually not at all what TrueTime guarantees and assuming they’ve solved a physical impossibility is dangerous technically as a founding assumption for higher level tech (which thankfully Spanner does not do). What TrueTime says is that clocks are synchronized within some delta just like NTP, but that delta is significantly smaller thanks to GPS time sync. That enables applications to have tighter bounds on wa…

That’s exactly what I’m saying but you simply provided more details. TrueTime guarantees clocks are well synchronized: and of course that means synchronized to a reasonable upper bound. It’s no more possible for clocks to be absolutely synchronized, than for two line segments drawn independently to have absolutely the same length.

> you can compare them to determine exactly which commit happened first

This is the part I was referring to. You cannot just compare timestamps and know which happened first. You have to actually handle the case where you don’t know if there’s a happens before relationship between the timestamps. Thats a very important distinction

Re: Clock synchronization is a nightmare

#39

Post by vlovich123 »

idorosen wrote: post #27
kccqzy wrote: post #25

Earlier quoted context omitted.

If you are an engineer at Google dealing with Spanner, then you can in fact assume clocks are well synchronized and can use timestamps for synchronization. If you get commit timestamps from Spanner you can compare them to determine exactly which commit happened first. That’s a stronger guarantee than the typical Serializable database like postgresql: https://www.postgresql.org/docs/current/transaction-iso.html... Tha…

Alternatively, you could guarantee the same synchronization using PPS and PTP to each host's DCD pin of their serial port or to specialized hardware such as modern PTP-enabled smart NICs/FPGAs that can accept PPS input. GPS+PPS gets you to within 20-80ns global synchronization depending on implementation (assuming you're all mostly in the same inertial frame), and allows you to make much stronger guarantees than True…

> and allows you to make much stronger guarantees than TrueTime (due to higher precision distributed ordering guarantees, which translate to lower latency and higher throughput distributed writes).

TrueTime is the software algorithm for managing the timestamps. It’s agnostic to the accuracy of the underlying time source. If it was inaccurate then you get looser bounds and as you note higher latency. Google already does everything you suggest for TrueTime while also having atomic clocks in places.

Re: Clock synchronization is a nightmare

#40

Post by vlovich123 »

awesome_dude wrote: post #26
kccqzy wrote: post #25

Earlier quoted context omitted.

If you are an engineer at Google dealing with Spanner, then you can in fact assume clocks are well synchronized and can use timestamps for synchronization. If you get commit timestamps from Spanner you can compare them to determine exactly which commit happened first. That’s a stronger guarantee than the typical Serializable database like postgresql: https://www.postgresql.org/docs/current/transaction-iso.html... Tha…

Isn't that because Google has its own atomic clocks, rather than NTP which is (generally) using publicly available atomic clocks?

More that they use GPS to synchronize the clocks. Having your own atomic clock doesn’t really improve your accuracy except for within the single data center you have it deployed (although I’m sure there’s techniques for synchronizing with low bounds against nearby atomic clocks + GPS to get really tight bound so they don’t need one in every data center)
Post reply on HN