Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clock synchronization is a nightmare

arpitbhayani.me

Post reply on HN
Print view View original

151–160 of 165 posts

Re: Clock synchronization is a nightmare

#151

Post by nuccy »

RossBencina wrote: post #89
nuccy wrote: post #65

Earlier quoted context omitted.

As a user of WhiteRabbit, I can confirm a sub-10ps sync (two clocks phase lock) over 50km fiber connection for variable temperature of fiber (biggest problem of clock sync over fibers is temperature induced length change of the fiber itself, which needs to be measured and compensated).

Out of interest, how do you measure a sub-10ps phase lock between devices 50km apart?

As of now, for testing, the two WR endpoints are sitting on the same desk with 50km fiber in a thermal chamber (simulating temperature changes in the soil), but in future they will be separated indeed.

Re: Clock synchronization is a nightmare

#152

Post by baby_souffle »

8n4vidtmkvmk wrote: post #118
baby_souffle wrote: post #91

Earlier quoted context omitted.

Not just audio, anybody in the live events / production space needs all equipment marching in lock step.

If it's for an event, can they not bring all the devices together in close proximity and sync them somehow? That at least removes network delays

> That at least removes network delays

But that's often _the source_ you need to work around; there is no way in hell that they're going to get all the light/sound/video/graphics ... etc people in the same 2 square meter area to put on a show like the super bowl :).

For smaller events like a touring act or even a venue with a few hundred people capacity you still need a single master clock but this time it's not "wall time" and is "absolute" time. E.g.: a musician at the front of the house chooses when to start and the video/lighting guy in the back needs to be on the same page so the visuals line up [0].

[0]: https://en.wikipedia.org/wiki/SMPTE_timecode

Re: Clock synchronization is a nightmare

#153

Post by sureshvoz »

This is a great breakdown, and it’s worth noting that we are hitting a "microsecond wall" in modern GPU clusters that makes standard NTP effectively obsolete.

In distributed training (LLMs), the bottleneck is no longer just disk I/O or CPU cycles—it’s the "straggler problem" during collective communication (like All-Reduce). When you’re running on 400Gbps+ RoCE (RDMA over Converged Ethernet) networks, the network "wire time" is often lower than the clock jitter on a standard Linux kernel.

If your clocks are skewed by even 2-3 milliseconds, your telemetry becomes essentially useless. It looks like packets are arriving before they were sent, or worse, your profiling tools can’t accurately pinpoint which GPU is stalling the rest of the 16,384-node fleet. We’ve reached a point where microsecond-accurate clocks isn't just a requirement for HFT firms; it’s becoming the baseline for anyone trying to keep $100s of millions of NVidia GPUs from idling while they wait for a collective sync.

Re: Clock synchronization is a nightmare

#154

Post by j_seigh »

The comments about HFT needing tightly synchronized clocks got me thinking.

Back in the day, way back in the 80's, IBM replaced the VM with VMXA. VM could trap and emulate all the important instructions since they were privileged instructions except one, the STCK (store clock) instruction. So virtual machines couldn't set their virtual clocks so they were always in sync. VMXA used new hw features that let you set the virtual clock. You could specify an offset to the system clock. But some of IBM's biggest customers depended on all the virtual machines clocks always being in sync. So VMXA had to add an option to disallow setting the clock for specified virtual machines.

Except all of development knew how trivial it was to trap or modify the STCK's to produce a timestamp of you choosing. This was before it was common knowledge the client code should never be trusted. But nobody enlightened IBM corporate management. It was a serious career limiting move at IBM. It didn't matter if you were right. So I'm pretty sure some serious fortunes were made as a result.

So the question for HFT is; are they using and trusting client timestamps, or are the timestamps being generated on the market maker's servers? If the latter, how would the customer know?

Re: Clock synchronization is a nightmare

#155

Post by jmpman »

Back in the early 2000s I was programming on an IBM AIX server. Multicore, maybe multiprocessor and within the same machine, the clocks were skewed between the processors. If you’d dispatch a process, and then check its outstanding running time, it would differ depending upon which processor you’d check from, and of course it was a signed type, and then we would get negative values, which sent our code down the wrong path.

Re: Clock synchronization is a nightmare

#156

Post by kobieps »

Dylan16807 wrote: post #109
kobieps wrote: post #10

Even just a single accurate clock is a nightmare... https://www.npr.org/2025/12/21/nx-s1-5651317/colorado-us-off...

I would not call "loses track of time if it's [partially] unplugged" a nightmare.

Haha fair, but in this case it was "loses the time if the power supply is interrupted"

Re: Clock synchronization is a nightmare

#157

Post by perryizgr8 »

sureshvoz wrote: post #153

This is a great breakdown, and it’s worth noting that we are hitting a "microsecond wall" in modern GPU clusters that makes standard NTP effectively obsolete. In distributed training (LLMs), the bottleneck is no longer just disk I/O or CPU cycles—it’s the "straggler problem" during collective communication (like All-Reduce). When you’re running on 400Gbps+ RoCE (RDMA over Converged Ethernet) networks, the network "wi…

If you have network infrastructure that supports 400G I'm pretty sure it has solid PTP built in. And as far as I remember from my networking days setting it up is almost as simple as setting up NTP, you just need a single machine with a GPS lock.

Re: Clock synchronization is a nightmare

#158

Post by Enginerrrd »

tobias2014 wrote: post #106
Enginerrrd wrote: post #72

Earlier quoted context omitted.

I think you just nerd-sniped me but I’m not convinced it’s impossible to assign a consistent ordering to events with relativistic separations. For starters, the spacetime interval between two events IS a Lorentz invariant quantity. That could probably be used to establish a universal order for timelike separations between events. I suspect that you could use a reference clock, like a pulsar or something to act as an…

For any space-like event you can find reference frames where things happen in different order. For the time-like situation you described the order indeed exists within the cone, which is to say that causality exists.

You can still order them with the spacetime interval compared to a reference event, even for space like separated events.

It allows for differing elements of the set to share the same value but so does using time alone. It just also allows every observer to agree on the ordering.

Bc Assigning a distance function to elements of a set is a common way to do that in fact. It doesn’t work with just a time coordinate or space coordinate, because that’s effectively a Euclidean metric.

You just have to contend with a few nonintuitive aspects but it’s not so bad.

Re: Clock synchronization is a nightmare

#159

Post by hinkley »

ianburrell wrote: post #47
emptybits wrote: post #43

Earlier quoted context omitted.

> But disconnected events can be seen in different orders depending on speed of observer. What are "disconnected events"? In a subtle but still real sense, are not all events causally linked? e.g. gravitationally, magnetically, subatomically or quantumly? I can understand that our simple minds and computational abilities lead us to consider events "far away" from each other as "disconnected" for practical reasons. Bu…

Causality moves at the speed of light. Events that are farther apart are called spacelike and aren't causally connected. I think you might be confusing events that have some history between them, and those are influence each other. Like say right now, Martian rover sends message to Earth and Earth sends message to them, those aren't causally connected cause don't know about the other message until light speed delay h…

We still haven’t proven whether some quantum effects do or don’t follow this. So there may be a loophole where information can move faster than light but the carrier for that information can not. Which might make ansibles possible some day, with the caveat that you can only have so many conversations per ansible before you need a refill with new entangled matter. In which case you have to divide the information by the travel time to determine your aggregate data rate. And the travel time will be at a fraction of the speed of light.

So you might be able to consult on which planet to terraform but you’re not going to video call the wife and kids unless you’re the richest person in the galaxy.

Re: Clock synchronization is a nightmare

#160

Post by ectospheno »

b112 wrote: post #117

The article doesn't cover the inane stupid that is: * NTP pool server usage requires using DNS * people have DNSSEC setup, which requires accurate time or it fails So if your clock is off, you cannot lookup NTP pool servers via DNS, and therefore cannot set your clock. This sheer stupid has been discussed with package maintainers of major distros, with ntpsec, and the result is a mere shrug. Often, the answer is "but…

I gave up and just got a gps ntp server. If you have a wood house then you can even get away with an inside antenna.
Post reply on HN