Open-sourcing a more precise time appliance • Hacker News Board

Re: Open-sourcing a more precise time appliance

#71

Post by cbhl » Wed, Aug 11, 2021, 2:32 PM UTC

michaelt wrote: post #18

A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their…

It looks like the source for their FBOSS agent that runs on switches will enable PTP in two kinds of supported switches:

https://github.com/facebook/fboss/blob/master/fboss/agent/hw...

They have O(thousands) of switches (source: https://engineering.fb.com/2016/10/18/data-center-engineerin...) so IIRC they get the per-switch cost down by doing a custom design to reduce BOM. I believe the specs and designs are available under the Open Compute project here (search for "Facebook"): https://www.opencompute.org/wiki/Networking/SpecsAndDesigns

Re: Open-sourcing a more precise time appliance

#72

Post by isodev » Wed, Aug 11, 2021, 2:35 PM UTC

Congrats to everyone who made this project possible. It’s certainly a great accomplishment and it solves a real world problem.

Now, does it require a Facebook account with my real name in order to use? /s

Re: Open-sourcing a more precise time appliance

#73

Post by BonoboIO » Wed, Aug 11, 2021, 2:39 PM UTC

latchkey wrote: post #58

A very long time ago , HP made a GPS time card just like this, encased in glass and filled with an inert gas such that if the card was tampered with, it was rendered useless. This prevented someone from faking the time signal.

Can you show us a picture. Did not find anything on google beside HP Laptop spam.

Re: Open-sourcing a more precise time appliance

#74

Post by judge2020 » Wed, Aug 11, 2021, 2:40 PM UTC

toomuchtodo wrote: post #67

NelsonMinar wrote: post #60

Man, tough crowd here. And someone who's been following precise timekeeping tech for 25 years now.. this is really cool! I think it's great some smart engineers have built a new reasonably compact and low cost very precise bit of timekeeping hardware. Thank you for sharing it open source!

I am torn. I absolutely love the technical blog post and the effort that went into open sourcing everything needed to build these devices, but, you know, Facebook. Compliments and respect to the engineers involved for really well done engineering.

Amazing people at Facebook make amazing things, but it does seem like only the worst of them (FB, Google) have the cash to burn to hire teams of engineers for projects outside of ones that directly contribute to generating revenue.

Re: Open-sourcing a more precise time appliance

#75

Post by Unklejoe » Wed, Aug 11, 2021, 2:43 PM UTC

rkangel wrote: post #4

I'm still unclear why they need the MAC on top of the GNSS reception. I can understand if it's just to provide holdover in event of GNSS signal loss, but they seem to be saying that it provides better precision when you've got both.

You use a jittery GPS PPS signal to discipline a highly stable primary clock through a very "slow" filter. The GPS PPS might have an RMS period jitter of say 10-20 nanoseconds, but the drift will be 0.

Then, you can use that disciplined primary clock to provide actual time for timestamping, etc.

Re: Open-sourcing a more precise time appliance

#76

Post by mlichvar » Wed, Aug 11, 2021, 2:43 PM UTC

abyagowi wrote: post #46

michaelt wrote: post #18

A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their…

Most (if not all) our switches in the datacenter are capable to perform as transparent clocks. With transparent clocks you can keep the time sync error down to less than 5ns per hop

Do the transparent clocks in your datacenter support unicast PTP?

There is a possibility to use PTP as a transport for NTP to take advantage of PTP-specific hardware timestamping. It could also process the correction field, but it seems the switches typically don't support unicast PTP.

The large asymmetry and banding of NTP in the test with Calnex Sentinel suggests it doesn't support the interleaved mode. NTP with hardware timestamping should normally be much more stable and symmetric, closer to PTP.

Re: Open-sourcing a more precise time appliance

#77

Post by michaelt » Wed, Aug 11, 2021, 2:43 PM UTC

mFixman wrote: post #62

michaelt wrote: post #18

A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their…

I'm curious: what's the use case for microsecond-precise timekeeping?

To monitor a control loop that ran in I could have used an in-between time unit, like hundredths of a millisecond, but that would have created embarrassingly long variable names.

Re: Open-sourcing a more precise time appliance

#78

Post by sophacles » Wed, Aug 11, 2021, 2:45 PM UTC

fanf2 wrote: post #59

michaelt wrote: post #39

Earlier quoted context omitted.

Although clock holdover might seem like a pointless feature - GPS outages are very unusual* - what holdover is mostly for is giving you time to respond if someone clumsily knocks your antenna over, or crushes your coax cable, or it gets damage that makes it fill with water, or something like that. A lot of the demand for high-precision clocks is for cell phone base stations, where there's no guarantee there'll be som…

Various militaries do regular GPS jamming exercises https://www.ofcom.org.uk/spectrum/information/gps-jamming-ex... https://www.navcen.uscg.gov/?pageName=gpsServiceInterruption...

GPS spoofing is also a thing. Here's a paper where they are able to spoof time while letting the receiver think it hasn't moved (much, within error bars): https://ieeexplore.ieee.org/document/6451170

That paper is in context of power grid equipment, but the GPS attack generalizes.

Re: Open-sourcing a more precise time appliance

#79

Post by Unklejoe » Wed, Aug 11, 2021, 2:45 PM UTC

willis936 wrote: post #25

hoseja wrote: post #10

Earlier quoted context omitted.

If I read correctly: GNSS PPS is more jittery but does not drift. The MAC has 1000x less jitter but drifts. You can cleverly combine them with statistics and magic to get the best of both worlds.

Phase-locked loops[0] are not magic ;) Also I imagine they are using a bog standard Kalman filter[1]. 0. https://en.wikipedia.org/wiki/Phase-locked_loop 1. https://en.wikipedia.org/wiki/Kalman_filter

I've implemented this (using GPS PPS to discipline a clock) using regular old PI control loops. I just make the loop damped enough to filter out the GPS PPS jitter and you're good.

Re: Open-sourcing a more precise time appliance

#80

Post by jeffbee » Wed, Aug 11, 2021, 2:56 PM UTC

cbhl wrote: post #71

michaelt wrote: post #18

A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their…

It looks like the source for their FBOSS agent that runs on switches will enable PTP in two kinds of supported switches: https://github.com/facebook/fboss/blob/master/fboss/agent/hw... They have O(thousands) of switches (source: https://engineering.fb.com/2016/10/18/data-center-engineerin... ) so IIRC they get the per-switch cost down by doing a custom design to reduce BOM. I believe the specs and designs are availab…

Even the retail price of 1 switch based on their "wedge" design is less than $10k for 32 100gbps ports. Merchant silicon like the Broadcom 56000-series switch ASICs support PTP.