Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a more precise time appliance

engineering.fb.com

Post reply on HN
Print view View original

161–170 of 188 posts

Re: Open-sourcing a more precise time appliance

#161

Post by ipsum2 »

adolph wrote: post #64

Seiko offers a GPS clock in a conventional-home friendly form factor. First introduced in 2013, the Seiko Space Link GPS clock has proven to be a global success. It is the world's first clock for home or office use that receives a time signal from a GPS satellite, thus delivering atomic clock precision. It measures time to within one second every 100,000 years. Equally importantly, it can be used anywhere in the worl…

This is a standard "atomic clock", that its not actually uses radioactive decay to accurately measure the passage of time, but receives GPS signals. The innovation in the post is using real "miniature atomic clock" that actually uses caesium 133.

Re: Open-sourcing a more precise time appliance

#162

Post by myself248 »

howenterprisey wrote: post #157
myself248 wrote: post #94

Earlier quoted context omitted.

It's not about accuracy (nobody really cares if the whole network is running fast or slow compared to some other clock) so much as synchrony (the whole network needs to be running from the same clock). Asynchronous networks can't guarantee latency or jitter performance, and they require buffers. Synchronous networks (remember, this is voice-grade stuff) can deliver bits out at the same rate as bits in, anywhere in th…

Cool, thanks for the writeup. Would you be able to suggest any books or other sources for further reading about this?

Nothing off the top of my head that's succinct and readable.

You'd do well to paw through the Bell Systems Technical Journal -- there's some amazing stuff in there like the origins of Unix and C [1], or Shannon's information theorem [2] -- but some details of the network itself are either hard to search for, or might not be there at all.

1: https://archive.org/details/bstj57-6-1991

2: https://archive.org/details/bstj27-4-623

Most of the old phreak philes are focused on the billing and routing aspects of the network, not so much on transport. I remember a few test equipment manuals had good introductions to the structure of a T1 frame, for instance, but perhaps not an exhaustive treatment of how that came to be.

This is an interesting question. It must be out there, mustn't it?

Re: Open-sourcing a more precise time appliance

#163

Post by bobsmooth »

yitchelle wrote: post #100
isodev wrote: post #72

Congrats to everyone who made this project possible. It’s certainly a great accomplishment and it solves a real world problem. Now, does it require a Facebook account with my real name in order to use? /s

What real world problem does this actually solve that has not been solved earlier? Genuinely curious.

Not having a low-cost extremely accurate Stratum 1 in a PCIe form-factor.

Re: Open-sourcing a more precise time appliance

#164

Post by kd913 »

the_only_law wrote: post #149
walrus01 wrote: post #148

> the ConnectX-6 Dx network card I would caution people against using the Mellanox/NVIDIA network interface cards in anything they care about. The driver license is proprietary and you have to build it for your kernel as a DKMS module in a semi manual process. Makes future system updates a real bother. I realize this probably doesn't apply so much to FB internally since they build and maintain their own distro in hou…

I’ve been wondering about SmartNICs recently and imagine all sorts of ideas for them. Which brand do you recommended? > Makes future system updates a real bother. Sounds like their GPU drivers too.

How about the x2 or Alveo series from Xilinx? It is basically the original definer of smartnic.

It comes from solarflare who have a long pedigree of low-latency smartnics. They used to supply Cloudflare, and also supply like 50% of fintechs/financial markets.

You can also just use openonload to accelerate your programs. In this case just doing straight linux socket programming, which can be accelerated without dpdk. Or just use the open source linux net driver if necessary.

https://github.com/Xilinx-CNS/onload

Re: Open-sourcing a more precise time appliance

#165

Post by pizza »

> The new service, built in-house and later open-sourced, was more scalable and improved the accuracy of timekeeping in the Facebook infrastructure from 10 milliseconds to 100 microseconds.

Interesting.. the infrastructure must therefore have a diameter of at most 30 km, thanks to the speed of light

Re: Open-sourcing a more precise time appliance

#166

Post by derefr »

m-ee wrote: post #158
PragmaticPulp wrote: post #143

Earlier quoted context omitted.

The Valve situation sounds like they attempted to open source after the fact, which doesn’t work with large hardware projects. The only way to really open source a hardware project of this magnitude is to set open source as a constraint from the start. If you don’t, you inevitably end up with someone licensing some code somewhere from a vendor that can’t be open sourced. It gets integrated deeply enough that the proj…

I shouldn't have called it open source, it was source available to customers who paid for the training course and signed a license agreement. Don't think they ever had intention of making it open. I found the model especially strange because of the requirement that any custom changes you asked for would be available to everyone else (your competitors). I guess they didn't actually want the workload of custom changes…

> I found the model especially strange because of the requirement that any custom changes you asked for would be available to everyone else (your competitors).

It's the same model as any game-console SDK. If you ask for changes to the SDK, those changes are going into "the SDK", which all customers get access to. Platform providers don't tend to offer work-for-hire for custom forks of their code; instead, what they're offering is to prioritize adding certain features to the shared upstream codebase. In the end, the platform provider wants there to just be one SDK, that solves the superset of all their customers' problems. Anything else is a ridiculously-large maintenance burden.

Re: Open-sourcing a more precise time appliance

#167

Post by tgot »

rkangel wrote: post #4

I'm still unclear why they need the MAC on top of the GNSS reception. I can understand if it's just to provide holdover in event of GNSS signal loss, but they seem to be saying that it provides better precision when you've got both.

The GPS can easily wobble by 50ns back and forth as the constellation changes. That's a lot! And, it is not random on a short time scale.

Folks often think "Oh, +/- 50ns, 20ns RMS, easy to filter...", but that's totally wrong.

The GPS will report -30ns from stable for minutes on end, then slew to +10ns, then -5ns, etc. Any high-precision oscillator (such as for radar) that's being jerked around like that isn't going to be as stable as high performance needs.

Even for just handoff of handsets at 2.2-2.3GHz, having the radio network (aka cell towers) all locked to an oven-controlled oscillator that was aligned-to, but far smoother-than, GPS, made a huge difference.

Now, improvements to GPS/GNSS that track 12 satellites instead of 6, and across multiple constellations, can result in more stable radio-based time. But then you get into urban canyons, and can only see 5 instead of 12, and you're right back into the jumpy situation.

Re: Open-sourcing a more precise time appliance

#168

Post by hinkley »

mFixman wrote: post #62
michaelt wrote: post #18

A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their…

I'm curious: what's the use case for microsecond-precise timekeeping?

There are consensus protocols that allow a grace period for clock skew, and the narrower you can constrain the clock skew, the more transactions per second you can retire, improving both scalability and latency.

Re: Open-sourcing a more precise time appliance

#169

Post by binkHN »

jrockway wrote: post #31

What's interesting to me is how the form factor of timing GPS modules has stayed constant over the years. I started my GNSS timing journey with a used Trimble GPS from the 2000s, and it has the same form factor and pinout as the modern multi-constellation timing GPSes from uBlox. I've had a GPS clock going for several years at this point, and without an atomic clock or really any fanciness (just LinuxPPS and Chrony),…

> I don't mind if my workstation is 20ms off UTC.

This is one of the reasons for OpenNTPD's existence. They wanted something that was simple and a ~50ms deviation was part of the design goals/trade off for this.

http://www.openntpd.org/goals.html

Re: Open-sourcing a more precise time appliance

#170

Post by LeifCarrotson »

anon_tor_12345 wrote: post #155
LeifCarrotson wrote: post #146

Earlier quoted context omitted.

It's a particular module for the Xilinx Artix 7 XC7A100T, made by a Shanghai-based integrator. As an individual user, you could buy these on Aliexpress, from eBay resellers, or from Amazon resellers. Facebook presumably has an arrangement with the contract manufacturer that's not going to be possible for an individual developer ordering ones and twos to replicate. However, to modify the HDL, you'd need a $3,000 licen…

you don't need vivado to modify the hdl, you need vivado (or some other proprietary xilinx toolchain) to produce the bitstream which actually deploys to the fpga.

That's a level of nitpicking that I didn't bother with.

Of course the Verilog or VHDL is just plaintext, which you can modify anywhere, but to actually do anything with it you need to compile it into a bitstream.

It does make me wonder if someone could set up a CI server to get around proprietary compilers like Vivado. Point the community resource at your source and get back a binary! Obviously you'd need to strip the license ID...

Post reply on HN