Live data from Hacker News

White Rabbit – sub-nanosecond synchronization for large distributed systems

ohwr.org

31–40 of 52 posts

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#31

What is significance of this?

Distributed systems spend most of their effort on one problem: agreeing on the order of events across machines. Without synchronized physical clocks you have two options. Logical clocks (Lamport, vector) give you causal order but not wall-clock truth, so you can’t answer “did A really happen before B” for events that don’t have a happens-before relationship. Or you run consensus, which gives total order but costs round trips. At geographic scale that’s tens of milliseconds per decision, and the floor is set by the speed of light.

Tight clock sync collapses this. If clock uncertainty ε is small and bounded, you can timestamp a write, wait ε, and trust the global order without talking to anyone. Spanner’s external consistency works because TrueTime’s ε was a few milliseconds, so commit-wait was tolerable. The latency cost of planet-scale serializability stops depending on how far apart your replicas are and starts depending on how good your clocks are.

That’s the real significance. Time sync converts a coordination problem (bounded by physics) into a local computation (bounded by clock quality). Spanner proved this is possible but required GPS receivers and atomic clocks in every datacenter, which kept the capability inside Google for years. White Rabbit-class sync pushes ε from milliseconds toward sub-nanoseconds over commodity Ethernet hardware, and it’s now in IEEE 1588 as a standard PTP profile. If sub-nanosecond sync becomes baseline network infrastructure, the long-held assumption that strong consistency has to be slow at geographic scale stops holding, and a meaningful chunk of what databases currently work around (HLCs, weak isolation defaults, application-level reconciliation) becomes unnecessary.

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#33
post #11

If you run "make" in the papers/IBIC2013 directory you'll get this paper: https://cds.cern.ch/record/1743073/files/thbl2.pdf It's quite interesting - this isn't ethernet as we know it. Instead of each NIC using its own free-running clock, all the physical layers are sync'ed to each other at layer 1. (note that gigabit ethernet, which is what it uses, sends data at all times - when idle it sends the idle symbol)

there are several 10 gbps implementations made in Europe https://gitlab.com/ohwr/project/10G-wr-nic/-/blob/f_ltgt_spe...

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#34
post #10

Earlier quoted context omitted.

Datacenters in spaaaace!

But they travel at 8 km/s so actually that cancels benefits? EDIT: checked, not enough to cancel them completely.

I wonder I'd that's the math for the ludicrous space data center ideas "floating" around...

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#35

What is significance of this?

Distributed systems spend most of their effort on one problem: agreeing on the order of events across machines. Without synchronized physical clocks you have two options. Logical clocks (Lamport, vector) give you causal order but not wall-clock truth, so you can’t answer “did A really happen before B” for events that don’t have a happens-before relationship. Or you run consensus, which gives total order but costs rou…

Very good explanation and interesting take on the 'humanity scale' or internet scale significance. I work on a phased array system so significance of white rabbit for me was always sample alignment. Assumed CERN had a similar use case of needing to order (sensor data of) physical events happening far apart.

But if we imagine the vast majority of internet and telecom infrastructure is also implemented this way, we can reason about information over time in general. Makes me think of 'earth is a big computer' type of sci fi trope. Neat!

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#36
If this wasn't CERN tech I would think I was being taken for a ride. Conventional wisdom is that distributed consensus is not possible at this kind of performance, does anyone have a sense for how this is different and how my mental model is wrong?

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#38

Earlier quoted context omitted.

But they travel at 8 km/s so actually that cancels benefits? EDIT: checked, not enough to cancel them completely.

I wonder I'd that's the math for the ludicrous space data center ideas "floating" around...

Just because you don't understand how something can work doesn't make it ludicrous. People like you are hell bent on destroying what's left of the Earth by turning it into a computer. If we left progress to those without an imagination, we wouldn't even have had a working calculator.

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#39
post #36

If this wasn't CERN tech I would think I was being taken for a ride. Conventional wisdom is that distributed consensus is not possible at this kind of performance, does anyone have a sense for how this is different and how my mental model is wrong?

> Conventional wisdom is that distributed consensus is not possible at this kind of performance

I'm not sure why you would think that? If you can assume the fiber is the same in both directions you know the round trip time is exactly double the latency of the connection. Then you know to phase shift your start time by that much when you get a start signal and you're in sync.

Obviously it's not trivial in practice, but it's not a fundamentally insurmountable problem.

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#40
post #14

Earlier quoted context omitted.

So then you need to know distance / roundtrip-length within centimeter precision as well (below 29.98 cm for sub-nanosecond precision… to be precise). Since cm precision is often not possible, is roundtrip-length an estimated average from prior roundtrips?

Hmm one would expect heat expansion to change the length of fiber over tens of kilometers. Does it also affect light speed in the fiber? I think consumer fiber is not buried very deep on average, but maybe for these use cases you use something hefty anyway.

It doesn't matter if the length changes provided:

* you measure the round trip time often enough

* the shift affects light in both directions equally

Post reply on HN