Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a more precise time appliance

engineering.fb.com

Post reply on HN
Print view View original

41–50 of 188 posts

Re: Open-sourcing a more precise time appliance

#41

Post by KaiserPro »

rkangel wrote: post #4

I'm still unclear why they need the MAC on top of the GNSS reception. I can understand if it's just to provide holdover in event of GNSS signal loss, but they seem to be saying that it provides better precision when you've got both.

Its more about stability in between the PPS from the GPS.

having the MAC means that the 10mhz reference signal is going to almost always be 10mhz, and in any given second contain 10million pulses[1].

However if you're just relying on GPS's PPS and a standard quartz oscillator, then the 10mhz reference is going to wobble about.

[1] this isn't really true, but illustrates the point of stability.

Re: Open-sourcing a more precise time appliance

#42

Post by the_biot »

nefitty wrote: post #2

I didn’t want to click this so I assume others here won’t either. Here’s the top points they highlight themselves: - Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure. - To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance. - With the help…

Whatever you may think of Facebook, they've done awesome things within their Open Compute Project. They also fund a lot of open source work.

Re: Open-sourcing a more precise time appliance

#43

Post by abyagowi »

MisterTea wrote: post #22

> While building our Time Appliance, we also invented a Time Card, a PCIe card that can turn any commodity server into a time appliance. Invented? These cards have been around. e.g. Meinberg makes a variety of expansion cards which use various time keeping sources such as WWVB, IRIG-B, GNSS, etc.

Please excuse the language “invented”. We simply meant “came up with a form factor” of having the MAC+GNSS present via PCIe bus as a POSIX clock similar to a NIC with hardware timestamping. We are humble to all the awesome products that are out there and we just wanted to present an open source version of a simplified approach to allow others to build on it.

Re: Open-sourcing a more precise time appliance

#44

Post by nonameiguess »

michaelt wrote: post #18

A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their…

I suspect Facebook data centers were already using $20,000 network switches that supported PTP.

Re: Open-sourcing a more precise time appliance

#45

Post by abyagowi »

tyingq wrote: post #29
MisterTea wrote: post #22

> While building our Time Appliance, we also invented a Time Card, a PCIe card that can turn any commodity server into a time appliance. Invented? These cards have been around. e.g. Meinberg makes a variety of expansion cards which use various time keeping sources such as WWVB, IRIG-B, GNSS, etc.

Those have the external source but don't seem to have the internal reference (Rubidium, Cesium, etc) for holdover. FB seems to be claiming to have invented a PCI card that makes a commodity server into a Stratum-1 source. Meinburg sells that too, but in proprietary appliance form.

Again, invented might be too strong. This is an abstraction of what you need minimally to build a time server on an open source basis. We appreciate and are humble to other products in the market

Re: Open-sourcing a more precise time appliance

#46

Post by abyagowi »

michaelt wrote: post #18

A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their…

Most (if not all) our switches in the datacenter are capable to perform as transparent clocks. With transparent clocks you can keep the time sync error down to less than 5ns per hop

Re: Open-sourcing a more precise time appliance

#47

Post by abyagowi »

nonameiguess wrote: post #40

> Now, as long as printing the PCB and soldering tiny components does not sound scary, anyone can build their own Time Card for a fraction of the cost of a regular time appliance. Doesn't sound scary, but not a lot of research labs and data centers are going to want to build their expansion cards from parts, even given free access to instructions. I understand if there isn't enough of a market for this to make it a v…

Of course we are not going to build the card in the lab with assembling SMD parts. This was more a way to explain that building such a card should not be out of access for anyone

Re: Open-sourcing a more precise time appliance

#48

Post by abyagowi »

mananaysiempre wrote: post #9
kd913 wrote: post #3

Why does Facebook need a precision time server?

To improve the performance of their distributed systems by imposing stronger assumptions than full asynchronicity on their environment, I expect. (See also Google’s Spanner.)

Of course we go inspired by the Google Spanner project and what they have published over a decade ago. Hats off to the Google Spanner team

Re: Open-sourcing a more precise time appliance

#49

Post by michaelt »

delayclose2 wrote: post #38

> The offset is practically 0 ranging within nanoseconds range. Uhm, no, the graph clearly shows that the offset is varying between 0-2 microseconds, which is 0 - 2,000 nanoseconds. 2,000 nanoseconds is not "practically zero". PTP precision is supposed to be in the 10ns range. Also, how did they get 40us precision on (what I assume to be) a normal kernel on a normal x86 box? I would have assumed that in a datacenter…

PTP is a standard feature on a lot of Intel chipsets so a lot of servers have it 'for free' and aren't using it.

It timestamps packets at the network card - independently of any kernel timing.

Re: Open-sourcing a more precise time appliance

#50

Post by efficax »

nefitty wrote: post #2

I didn’t want to click this so I assume others here won’t either. Here’s the top points they highlight themselves: - Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure. - To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance. - With the help…

Oh no if you click it Facebook might find out! The horror
Post reply on HN