Live data from Hacker News

Clock synchronization is a nightmare

arpitbhayani.me

51–60 of 165 posts

Re: Clock synchronization is a nightmare

#51

On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at ho…

At this point the only clock in my life that doesn't auto set is the one on my stove, and that's because I abhor internet connected kitchen appliances.

Re: Clock synchronization is a nightmare

#52
post #3

PTP requires support not only on your network, but also on your peripheral bus and inside your CPU. It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states.

You can if you just run PTP (almost) entirely on your NIC. The best PTP implementations take their packet timestamps at the MAC on the NIC and keep time based on that. Nothing about CPU processing is time-critical in that case.

Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. The control loop for host PTP benefits from regularity. Anyway NICs that support PTP hardware timestamping may also use PCI LTR (latency tolerance reporting) to instruct the host operating system to disable high-exit-latency sleep features, and popular operating systems respect that.

Re: Clock synchronization is a nightmare

#53
> When two transactions happen at nearly the same time on different nodes, the database must determine which happened first. If clocks are out of sync, the database might order them incorrectly, violating consistency guarantees.

This is only true if you use wall clock time as part of your database’s consistency algorithm. Generally I think this is a huge mistake. It’s almost always much easier to swap to a logical clock - which doesn’t care about wall time. And then you don’t have to worry about ntp.

The basic idea is this: event A happened before event B iff A (or something that happened after A) was observed by the node that generated B before B was generated. As a result, you end up with a dag of events - kind of like git. Some events aren’t ordered relative to one another. (We say, they happened concurrently). If you ever need a global order for all events, you can deterministically pick an arbitrary order for concurrent events by comparing ids or something. And this will give you a total order that will be the same on all peers.

If you make database events work like this, time is a little more complex. (It’s a graph traversal rather than simple numbers). But as a result the system clock doesn’t matter. No need to worry about atomic clocks, skew, drift, monotonicity, and all of that junk. It massively simplifies your system design.

Re: Clock synchronization is a nightmare

#54
post #49

Earlier quoted context omitted.

Isn't it also useful for checking validity periods for stuff like TLS certs or JWTs or Kerberos tickets?

We could use some made up „time“ for that since it’s not made for human consumption just sync between different systems.

I suppose this is changing with TLS certs moving towards ephemerality, but they used to have an entry on someone's calendar for renewal.

Re: Clock synchronization is a nightmare

#55

Unfortunate that the author doesn’t bring up FoundationDB version stamps, which to me feel like the right solution to the problem. Essentially, you can write a value you can’t read until after the transaction is committed and the synchronization infrastructure guarantees that value ends up being monotonically increasing per transaction. They use similar “write only” operations for atomic operations like increment.

Yes. A consistent total ordering is what you need (want) in distributed computing. Ultimately, causality is what is important, but consistent ordering of concurrent operations makes things much easier to work with.

Consistent ordering of concurrent operations is easy though. Just detect this case (via logical clocks) then order using node ids or transaction ids if the logical clocks show the transactions as being concurrent. Am I missing something? This feels like a very solved problem. (I’ve worked on CRDTs where we have the same problem. There exist incredibly fast algorithms for this.)

Re: Clock synchronization is a nightmare

#56
post #23

Ok,so people use NTP to "synchronize" their clocks and then write applications that assume the clocks are in exact sync and can use timestamps for synchronization, even though NTP can see the clocks aren't always in sync. Do I have that right?

*misuse timestamps for synchronization

Re: Clock synchronization is a nightmare

#57

On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at ho…

At this point the only clock in my life that doesn't auto set is the one on my stove, and that's because I abhor internet connected kitchen appliances.

Same here. I wish there was an easy way around it (that doesn't require me to play sysadmin in my spare time).

Re: Clock synchronization is a nightmare

#58

Earlier quoted context omitted.

There is distinction between seeing when events happened, and when they really happened. The latter can be reconstructed by an observer. In special relativity, time is relative and when things actually happened can be different in different frames. Casually linked events are always really in the same order. But disconnected events can be seen in different orders depending on speed of observer.

> But disconnected events can be seen in different orders depending on speed of observer. What are "disconnected events"? In a subtle but still real sense, are not all events causally linked? e.g. gravitationally, magnetically, subatomically or quantumly? I can understand that our simple minds and computational abilities lead us to consider events "far away" from each other as "disconnected" for practical reasons. Bu…

> But wouldn't these causally disjoint regions overlap with each other

Yes.

> stringing together a chain of causality from anywhere to anywhere?

No? Causality reaching one edge of a sphere doesn't mean it instantaneously teleports to every point in that same sphere. This isn't a transitive relationship.

> What are "disconnected events"?

The sentence you're responding to seems like a decent definition. Disconnected events are events which might be observed in either order depending on the position of an observer.

Re: Clock synchronization is a nightmare

#59
post #25

Earlier quoted context omitted.

If you are an engineer at Google dealing with Spanner, then you can in fact assume clocks are well synchronized and can use timestamps for synchronization. If you get commit timestamps from Spanner you can compare them to determine exactly which commit happened first. That’s a stronger guarantee than the typical Serializable database like postgresql: https://www.postgresql.org/docs/current/transaction-iso.html... Tha…

Alternatively, you could guarantee the same synchronization using PPS and PTP to each host's DCD pin of their serial port or to specialized hardware such as modern PTP-enabled smart NICs/FPGAs that can accept PPS input. GPS+PPS gets you to within 20-80ns global synchronization depending on implementation (assuming you're all mostly in the same inertial frame), and allows you to make much stronger guarantees than True…

Of course, you can do this in good conditions. The extremely powerful part that TrueTime brings is how the system degrades when something goes wrong.

If everyone is synced to +/- 20ns, that's great. Then when someone flies over your datacenter with an GPS jammer (purposeful or accidental), this needs to not be a bad day where suddenly database transactions happen out of order, or you have an outage.

The other benefit of building in this uncertainty to the underlying software design is you don't have to have your entire infrastructure on the same hardware stack. If you have one datacenter that's 20yrs old, has no GPS infrastructure, and operates purely on NTP - this can still run the same software, just much more slowly. You might even keep some of this around for testing - and now you have ongoing data showing what will happen to your distributed system if GPS were to go away in a chunk of the world for some sustained period of time.

And in a brighter future, if we're able to synchronize everyone's clocks to +/- 1ns, the intervals just get smaller and we see improved performance without having to rethink the entire design.

Re: Clock synchronization is a nightmare

#60
post #57

Earlier quoted context omitted.

At this point the only clock in my life that doesn't auto set is the one on my stove, and that's because I abhor internet connected kitchen appliances.

Same here. I wish there was an easy way around it (that doesn't require me to play sysadmin in my spare time).

In the 80s my uncle had digital clocks that used an antenna to tune into the atomic clock time signal that (was/is?) broadcast nationwide. I've long wished that it was incorporated into stoves, microwaves, essentially everything that isn't an internet device (yet... sigh)

Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of a pin these days.

Post reply on HN