Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clock synchronization is a nightmare

arpitbhayani.me

Post reply on HN
Print view View original

111–120 of 165 posts

Re: Clock synchronization is a nightmare

#111

Post by jasonwatkinspdx »

namibj wrote: post #99
jasonwatkinspdx wrote: post #68

Earlier quoted context omitted.

> Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of a pin these days. Unfortunately there's no real way to cheat physics as far as shrinking a wavelength goes. With RF antennas about the best you can do is a major dimension 1/10th the frequency of interest.

There are many DCF77 receivers in Germany that are contained in a square box that's barely large enough for a AA battery; the rest of the square contains the motor/gears and the electronics/receiver (incl. a ferrite loopstick antenna). The wavelength is around 3.8km...

Yeah, that's because it's receiving an extremely narrowband signal accumulated over a long window so it can suffer the trash efficiency I'm talking about.

Re: Clock synchronization is a nightmare

#112

Post by ssl-3 »

Dylan16807 wrote: post #108
ssl-3 wrote: post #95

Earlier quoted context omitted.

Both of the WWVB clocks I've owned have been very fickle about how they're placed because RF be that way sometimes, and Colorado isn't exactly nearby to my location in Ohio. The first manufactured GPS clock I owned (as in: switch it on and time is shown on a dedicated display) was in a 2007 Honda. But a firmware bug ruined that clock: https://didhondafixtheclocks.com/ And even after it began displaying the right time…

On the one hand, some sloppy GPS units fail on a 20 year schedule. On the other hand, a bunch of things using NTP are going to fail in about ten years. (2036 rather than 2038 because reasons)

Yeah, good point.

If I ever get the chance, I'll try to remember to tell the 1995 version of me to watch out for that pesky overflow bug that they might experience with NTP -- two score and 1 year in their future.

Re: Clock synchronization is a nightmare

#113

Post by Dylan16807 »

jeffbee wrote: post #52
pclmulqdq wrote: post #4

Earlier quoted context omitted.

You can if you just run PTP (almost) entirely on your NIC. The best PTP implementations take their packet timestamps at the MAC on the NIC and keep time based on that. Nothing about CPU processing is time-critical in that case.

Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. The control loop for host PTP benefits from regularity. Anyway NICs that support PTP hardware timestamping may also use PCI LTR (latency tolerance reporting) to instruct the host operating system to disable high-exit-latency sleep features, and popular operating systems respect that.

> The control loop for host PTP benefits from regularity.

How much regularity? If you sent PTP packets with 5 milliseconds of randomness in the scheduling, does that cause real problems? It's still going to have an accurate timestamp.

> instruct the host operating system to disable high-exit-latency sleep features

Why, though? You didn't explain this. As long as the packet got timestamped when it arrived, the CPU can ask the NIC how many nanoseconds ago that was, and correct for how long it was asleep. Right?

Re: Clock synchronization is a nightmare

#114

Post by toast0 »

vel0city wrote: post #103
toast0 wrote: post #98

Earlier quoted context omitted.

It's hard to keep a phone's clock closely synchronized because they experience a lot of temperature swings, going between pockets and hands and open air and sometimes in direct sun, and the processor goes between idle and 100% as well. Once you get to internationa phones, you'll have places where the phone does not include all timezones and specifically is missing the actual local timezone, so automatic sync is typic…

Don't a lot of cellular networks rely on highly synchronized clocks to properly handle TDMA-style transmissions? Shouldn't they be very in sync with the towers' times?

I do believe there are (fairly) tight tolerances for clock synchronization between the network and the user equipment/handsets, but I don't know that it necessarily involves communicating the wallclock time. And the oscillators for signal timing aren't necessarily used for timekeeping.

Re: Clock synchronization is a nightmare

#115

Post by christina97 »

toast0 wrote: post #98
Uehreka wrote: post #77

Earlier quoted context omitted.

I was doing something at work that involved calculating round trip times from/to Android devices, and learned that although it should be possible for NTP to sync clocks with below-second precision, in practice many of the Android devices I was working with (mostly Pixels 2-7) were off from my server and each other by up to 5 seconds, which blew my mind.

It's hard to keep a phone's clock closely synchronized because they experience a lot of temperature swings, going between pockets and hands and open air and sometimes in direct sun, and the processor goes between idle and 100% as well. Once you get to internationa phones, you'll have places where the phone does not include all timezones and specifically is missing the actual local timezone, so automatic sync is typic…

It’s not that hard. You would not expect 5 sec drift on phones that can sync time on the web at least once a day or once a week. A basic quartz crystal can keep time to within seconds per month of drift. High quality phones can do the same or better. Also the phone should keep track of system time as epoch time, and convert to local.

Re: Clock synchronization is a nightmare

#116

Post by vlovich123 »

idorosen wrote: post #62
vlovich123 wrote: post #39

Earlier quoted context omitted.

> and allows you to make much stronger guarantees than TrueTime (due to higher precision distributed ordering guarantees, which translate to lower latency and higher throughput distributed writes). TrueTime is the software algorithm for managing the timestamps. It’s agnostic to the accuracy of the underlying time source. If it was inaccurate then you get looser bounds and as you note higher latency. Google already do…

Yup! I was referring to the original TrueTime/Spanner papers, not whatever's currently deployed. The original paper makes reference to distributed ordering guarantees at the milliseconds' scale precision, which implies many more transactions in flight in the uncertain state and coarser distributed ordering guarantees than the much tighter upper bound you can set with nanoseconds' precision and microseconds' comms lat…

More than a decade of progress, probably in no small part from Google pushing vendors to improve hardware :)

Re: Clock synchronization is a nightmare

#117

Post by b112 »

The article doesn't cover the inane stupid that is:

* NTP pool server usage requires using DNS

* people have DNSSEC setup, which requires accurate time or it fails

So if your clock is off, you cannot lookup NTP pool servers via DNS, and therefore cannot set your clock.

This sheer stupid has been discussed with package maintainers of major distros, with ntpsec, and the result is a mere shrug. Often, the answer is "but doesn't your device have a battery backed clock?", which is quite unhelpful. Many devices (routers, IOT devices, small boards, or older machines, etc) don't have a battery backed clock, or alternatively the battery may just have died.

Beyond that, the ntpsec codebase has a horrible bug where if DNS is not available when ntpsec starts, pool server addresses are never, ever retried. So if you have a complete power-fail in a datacentre rack, and your firewalls take a little longer to boot than your machines, you'll have to manually restart ntpsec to even get it to ever sync.

When discussing this bug the ntpsec lads were confused that DNS might not exist at times.

Long story short, make sure you aren't using DNS in any capacity, in NTP configs, and most especially in ntpsec configs.

One good source is just using the IPs provided by NIST. Pool servers may seem fine, but I'd trust IPs assigned to NIST to exist longer than any DNS anyhow. EG, for decades.

Re: Clock synchronization is a nightmare

#118

Post by 8n4vidtmkvmk »

baby_souffle wrote: post #91
dmazin wrote: post #73

I highly recommend anyone to look up how PTP works and how it compares to NTP. Clock sync is very interesting. When I joined an HFT company, first thing I did was understand this stuff. We care about it a lot[1]. If you want a specific question to answer, answer this: why does PTP need hardware timestamping to achieve high precision (where the network card itself assigns timestamps to packets, rather than having the…

Not just audio, anybody in the live events / production space needs all equipment marching in lock step.

If it's for an event, can they not bring all the devices together in close proximity and sync them somehow? That at least removes network delays

Re: Clock synchronization is a nightmare

#119

Post by b112 »

josephg wrote: post #53

> When two transactions happen at nearly the same time on different nodes, the database must determine which happened first. If clocks are out of sync, the database might order them incorrectly, violating consistency guarantees. This is only true if you use wall clock time as part of your database’s consistency algorithm. Generally I think this is a huge mistake. It’s almost always much easier to swap to a logical cl…

Unfortunately, some of us have to deal with things like billing, transaction timing to validate what a client's logs might have on their systems, and so on.

My take on this is that second timing is close enough for this. And that all my internal systems need agree on the time. So if I'm off by 200ms or some blather from the rest of the world, I'm not overly concerned. I am concerned, however, if a random internal system is not synced to my own ntp servers.

This doesn't mean I don't keep our servers synced, just that being off by some manner of ms doesn't bother me inordinately. And when it comes to timing of events, yes, auto-increment IDs or some such are easier to deal with.

Re: Clock synchronization is a nightmare

#120

Post by idorosen »

vlovich123 wrote: post #116
idorosen wrote: post #62

Earlier quoted context omitted.

Yup! I was referring to the original TrueTime/Spanner papers, not whatever's currently deployed. The original paper makes reference to distributed ordering guarantees at the milliseconds' scale precision, which implies many more transactions in flight in the uncertain state and coarser distributed ordering guarantees than the much tighter upper bound you can set with nanoseconds' precision and microseconds' comms lat…

More than a decade of progress, probably in no small part from Google pushing vendors to improve hardware :)

Amen. :)
Post reply on HN