How Precision Time Protocol is being deployed at Meta • Hacker News Board

Re: How Precision Time Protocol is being deployed at Meta

#21

Post by ElevenLathe » Mon, Nov 21, 2022, 7:16 PM UTC

throw0101a wrote: post #9

CapmCrackaWaka wrote: post #6

Why do so many tech companies seem to be releasing “secret sauce” for free lately? I see a lot of posts lately detailing how inner production systems work at large companies, and while I’m grateful, I’m curious why the higher ups think it’s worthwhile to release this information.

> Why do so many tech companies seem to be releasing “secret sauce” for free lately? Is it really their "special sauce" though? Do these types of releases actually give away how these companies make money ? In this particular case, telling the world how to get to nanosecond levels of timekeeping doesn't really help any competitors take away Metabook's revenues or profits.

OTOH if there turns out to be some other way of making money on nanosecond timekeeping, then keeping it secret will help them get into that market before competitors can.

Re: How Precision Time Protocol is being deployed at Meta

#22

Post by waynesonfire » Mon, Nov 21, 2022, 7:18 PM UTC

I remember Google doing this many years ago, does anyone recall the papers that describes distributed storage systems that benefit from highly accurate clocks? Was it spanner or the next generation version of it? I can't remember. I recall finding it fascinating but out or reach since I think they were putting hardware based atomic clocks on their systems, sounded super proprietary.

Re: How Precision Time Protocol is being deployed at Meta

#23

Post by jasonwatkinspdx » Mon, Nov 21, 2022, 7:20 PM UTC

klodolph wrote: post #15

throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

I would also love to see an explanation of “why do we need this much accuracy?” that actually goes through the derivation of how much accuracy you need. Some of the justification for Google’s TrueTime is found in the Spanner docs: https://cloud.google.com/spanner/docs/true-time-external-con... Basically, you want to be able to do a “snapshot read” of the database rather than acquiring a lock (for reasons which should…

Meta uses some variations on Hybrid Logical Clocks, which are very similar to TrueTime, so yes this does apply. Besides performance they very much want to avoid consistency issues that could result in a security breach, eg, if I block Alan and then post "Alan is a dookie head" you don't want some node seeing the second event first. Well really the bigger concern is someone spots this as a potential vulnerability point and scripts something.

Re: How Precision Time Protocol is being deployed at Meta

#24

Post by anonymousDan » Mon, Nov 21, 2022, 7:26 PM UTC

waynesonfire wrote: post #22

I remember Google doing this many years ago, does anyone recall the papers that describes distributed storage systems that benefit from highly accurate clocks? Was it spanner or the next generation version of it? I can't remember. I recall finding it fascinating but out or reach since I think they were putting hardware based atomic clocks on their systems, sounded super proprietary.

Yeah, Spanner and the Truetime API. Would be interesting to know whether there are many differences between the implementations (or indeed the abstractions offered). I always wonder whether there are rare occasions where such systems fail (e.g. the bounds reported don't actually capture the true time due to some bug), and what the resulting implications are for systems built on top of them.

Re: How Precision Time Protocol is being deployed at Meta

#25

Post by cheriot » Mon, Nov 21, 2022, 7:33 PM UTC

zzixp wrote: post #16

I've been seeing a lot more of these Meta engineering articles on HN - I'm a fan! If you're writing them, keep up the great work!

Agreed. I wonder if the motivation is competing for resources with Meta paying more attention to costs now.

Re: How Precision Time Protocol is being deployed at Meta

#26

Post by willis936 » Mon, Nov 21, 2022, 7:34 PM UTC

kazinator wrote: post #3

It is not accurate to say that PTP is a predecessor to NTP. You need both. PTP synchronizes clocks to a ridiculous precision, like down to nanoseconds. To do that, it uses support in the ethernet hardware. Hardware adds precision stamps to the PTP-related ethernet frames, so that the time calculations are free of jitters induced by the higher network stacks. A cool thing is that PTP-aware routers/switches can actuall…

Just because high precision tasks don't care about time of day as much, PTP still uses TAI timestamps. The only time of day information missing is leap seconds, and those were a mistake that is being corrected. You don't need NTP for time of day when PTP is used.

Re: How Precision Time Protocol is being deployed at Meta

#27

Post by RockRobotRock » Mon, Nov 21, 2022, 7:38 PM UTC

About a year ago, Linus from LTT collaborated with the Facebook team to make a video about the subject: https://youtu.be/JK3eTGkX6qY

Very cool!

Re: How Precision Time Protocol is being deployed at Meta

#28

Post by xboxnolifes » Mon, Nov 21, 2022, 7:48 PM UTC

ElevenLathe wrote: post #21

throw0101a wrote: post #9

Earlier quoted context omitted.

> Why do so many tech companies seem to be releasing “secret sauce” for free lately? Is it really their "special sauce" though? Do these types of releases actually give away how these companies make money ? In this particular case, telling the world how to get to nanosecond levels of timekeeping doesn't really help any competitors take away Metabook's revenues or profits.

OTOH if there turns out to be some other way of making money on nanosecond timekeeping, then keeping it secret will help them get into that market before competitors can.

This is just spending "potential money" for engineering talent advertisements.

Re: How Precision Time Protocol is being deployed at Meta

#29

Post by milleramp » Mon, Nov 21, 2022, 7:55 PM UTC

Companies have been using PTP to synchronize (frame sync) networked cameras for many years. It is much better than wiring up a separate ttl or differential signals. It is amazing the accuracy you can get with this protocol.

Re: How Precision Time Protocol is being deployed at Meta

#30

Post by madsmith » Mon, Nov 21, 2022, 8:08 PM UTC

CapmCrackaWaka wrote: post #6

Why do so many tech companies seem to be releasing “secret sauce” for free lately? I see a lot of posts lately detailing how inner production systems work at large companies, and while I’m grateful, I’m curious why the higher ups think it’s worthwhile to release this information.

1) it makes the engineers happy to be able to speak about the interesting projects they are working on and give back to the wider community.

2) it serves as a venue for attracting other talented candidates who are likewise minded on working on technical problems.

3) when I was employed of FB it was a relatively flat hierarchy, which is to say there weren’t that many higher ups to convince that this should be done.