Live data from Hacker News

White Rabbit – sub-nanosecond synchronization for large distributed systems

ohwr.org

41–50 of 52 posts

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#41
If your system depends on highly accurate time maybe you’re doing it wrong?

Of course no doubt there’s some requirements that cannot avoid that requirement.

But for the most part needing synchronized time is just going to be a forever problem isn’t it?

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#42

What is significance of this?

Distributed systems spend most of their effort on one problem: agreeing on the order of events across machines. Without synchronized physical clocks you have two options. Logical clocks (Lamport, vector) give you causal order but not wall-clock truth, so you can’t answer “did A really happen before B” for events that don’t have a happens-before relationship. Or you run consensus, which gives total order but costs rou…

Awesome explainer, thanks for that

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#43
post #29
post #14

Earlier quoted context omitted.

So then you need to know distance / roundtrip-length within centimeter precision as well (below 29.98 cm for sub-nanosecond precision… to be precise). Since cm precision is often not possible, is roundtrip-length an estimated average from prior roundtrips?

... why would cm precision be often not possible?

Because you're using fibres laid years ago. Often pulse velocity isn't known to better than one part in 1000.

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#44

Earlier quoted context omitted.

I wonder I'd that's the math for the ludicrous space data center ideas "floating" around...

Just because you don't understand how something can work doesn't make it ludicrous. People like you are hell bent on destroying what's left of the Earth by turning it into a computer. If we left progress to those without an imagination, we wouldn't even have had a working calculator.

That might be true if the proponents had an argument of any sort in support of their plan, other than "we need idiots to give us money".

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#45

Earlier quoted context omitted.

I wonder I'd that's the math for the ludicrous space data center ideas "floating" around...

Just because you don't understand how something can work doesn't make it ludicrous. People like you are hell bent on destroying what's left of the Earth by turning it into a computer. If we left progress to those without an imagination, we wouldn't even have had a working calculator.

You're right.

It's the physics of cooling the beasts and the communication delays that make those plans ludicrous.

To turn your assertion on its head, the fact that the supporters don't seem to be able (or willing) to do the math to fact check these proposals is not an indicator that the plans will work.

As a starting point for comparison, the total power budget of the ISS is under 100kW and a single supercomputer rack dissipates about 4x that. What changes to the ISS can be made to get 100x more power and dissipate 100x more heat?

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#46
post #39
post #36

If this wasn't CERN tech I would think I was being taken for a ride. Conventional wisdom is that distributed consensus is not possible at this kind of performance, does anyone have a sense for how this is different and how my mental model is wrong?

> Conventional wisdom is that distributed consensus is not possible at this kind of performance I'm not sure why you would think that? If you can assume the fiber is the same in both directions you know the round trip time is exactly double the latency of the connection. Then you know to phase shift your start time by that much when you get a start signal and you're in sync. Obviously it's not trivial in practice, bu…

Twice the path delay + the time it takes to send the return packet. I assume WR does this in hardware to get a predictable time.

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#47

Earlier quoted context omitted.

Distributed systems spend most of their effort on one problem: agreeing on the order of events across machines. Without synchronized physical clocks you have two options. Logical clocks (Lamport, vector) give you causal order but not wall-clock truth, so you can’t answer “did A really happen before B” for events that don’t have a happens-before relationship. Or you run consensus, which gives total order but costs rou…

Very good explanation and interesting take on the 'humanity scale' or internet scale significance. I work on a phased array system so significance of white rabbit for me was always sample alignment. Assumed CERN had a similar use case of needing to order (sensor data of) physical events happening far apart. But if we imagine the vast majority of internet and telecom infrastructure is also implemented this way, we can…

Indeed, time synchronization across detectors is always tricky. Distributed clocks get messy at ATLAS dimensions. WR allows to distribute pretty good time sync over large detector systems. Sometimes still not good enough though. Time-of-flight detectors try to get to single-digit ps level, and almost by definition, you have to synchronize two detectors that are some distance apart.

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#48

What is significance of this?

Distributed systems spend most of their effort on one problem: agreeing on the order of events across machines. Without synchronized physical clocks you have two options. Logical clocks (Lamport, vector) give you causal order but not wall-clock truth, so you can’t answer “did A really happen before B” for events that don’t have a happens-before relationship. Or you run consensus, which gives total order but costs rou…

I don’t see why sub-nanosecond sync is useful for Spanner-style ordering. Your average server is more than 1 light-ns wide! Your average cable from server to TOR switch is several light-ns long!

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#49
post #2

Haven't looked into this in depth but sub-nanosecond sync for systems up to 10km apart is interesting since 10km is about 33 light microseconds. There is some trickery going on.

If both systems have a good clock. Then the synchronization messages only need to contain the time delta to correct the time (phase?) drift to achieve full synchronization.

Re: White Rabbit – sub-nanosecond synchronization for large distributed systems

#50

Earlier quoted context omitted.

Just because you don't understand how something can work doesn't make it ludicrous. People like you are hell bent on destroying what's left of the Earth by turning it into a computer. If we left progress to those without an imagination, we wouldn't even have had a working calculator.

You're right. It's the physics of cooling the beasts and the communication delays that make those plans ludicrous. To turn your assertion on its head, the fact that the supporters don't seem to be able (or willing) to do the math to fact check these proposals is not an indicator that the plans will work. As a starting point for comparison, the total power budget of the ISS is under 100kW and a single supercomputer ra…

Oh I have done the math. There are multiple ways to get cooling to work in space:

1. There is no real size limit to radiators in space, especially when in solar orbit.

2. High temperature chip architectures can be used, operating at 600K.

3. Heat pipes can bring the temperature even higher, such as to 1200K.

4. Special 3D radiator geometries can be used to optimize heat escape.

5. Metamaterials can be used to optimize photon emission in the best directions.

Together, these will shrink the required radiator area dramatically. Beyond these standard ideas, other exotic approaches exist at the edge of viability.

The ISS in contrast is restricted considering it has to sit in Earth orbit.

You may see this video (AI researched and generated) to grasp some of the points: https://www.youtube.com/watch?v=s7Mv_OcBXI8

Post reply on HN