Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Precision Time Protocol is being deployed at Meta

engineering.fb.com

Post reply on HN
Print view View original

11–20 of 104 posts

Re: How Precision Time Protocol is being deployed at Meta

#11

Post by BaconPackets »

throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

I would be really curious to drill down into one of these problems. Is super precise really the solution?

Re: How Precision Time Protocol is being deployed at Meta

#12

Post by neilv »

CapmCrackaWaka wrote: post #6

Why do so many tech companies seem to be releasing “secret sauce” for free lately? I see a lot of posts lately detailing how inner production systems work at large companies, and while I’m grateful, I’m curious why the higher ups think it’s worthwhile to release this information.

Maybe it's the usual reasons that we see companies releasing stuff, not only secret sauce?

I think usually it's for company PR for various purposes (counteract bad press, attract new hires, etc.).

Sometimes to generate a bigger hiring pool that knows the stuff you're releasing. (And the open source story about crowdsourcing contributions, which sometimes might be worth the costs.)

I've also seen it around partnerships and customer collaborations and competition. Including to "commoditize your complement", or to kill one thing with what they'd rather use. (And, in industry/tech standards, corporate representatives often have motivation to try to bias the standard to their employer.)

In some cases, it's for individual employees' careers. Think how academic and some R&D jobs want research publications, or how some companies want people who do "talks".

Sometimes also for getting code/docs public, so employees can still use it when they leave.

Re: How Precision Time Protocol is being deployed at Meta

#13

Post by EwanToo »

CapmCrackaWaka wrote: post #6

Why do so many tech companies seem to be releasing “secret sauce” for free lately? I see a lot of posts lately detailing how inner production systems work at large companies, and while I’m grateful, I’m curious why the higher ups think it’s worthwhile to release this information.

The question for 2023 is "How many companies will be investing in this without a clear revenue stream?"

It's quite likely we're entering a period where the current baseline performance of core infrastructure will be considered "good enough" and companies won't employ people to work on these general improvements.

Re: How Precision Time Protocol is being deployed at Meta

#14

Post by jtsiskin »

epgui wrote: post #2

This is incredible. If I knew I'd be working on things like this (as opposed to optimizing ad revenue or "user engagement" by A/B testing button copy), I'd possibly consider working at Meta.

At Meta you choose what team you want to work on during a 3 month “bootcamp” period, very different from Google, Amazon, etc. There’s enough teams solving hard distributed system problems that you’re guaranteed to never need to think about button copy

Re: How Precision Time Protocol is being deployed at Meta

#15

Post by klodolph »

throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

I would also love to see an explanation of “why do we need this much accuracy?” that actually goes through the derivation of how much accuracy you need.

Some of the justification for Google’s TrueTime is found in the Spanner docs:

https://cloud.google.com/spanner/docs/true-time-external-con...

Basically, you want to be able to do a “snapshot read” of the database rather than acquiring a lock (for reasons which should be apparent). The snapshot read is based on a monotonic clock. You can get much better performance out of your monotonic clock if all of your machines have very accurate clocks. When you write to the database, you can add a timestamp to the operation, but you may have to introduce a delay to account for the worst-case error in the clock you used to generate the timestamp.

More accurate timestamps -> less delay. From my understanding, less delay -> servers have more capacity -> buy fewer servers -> save millions of dollars -> use savings to pay for salaries of people who figured out how to make super precise timestamps and still come out ahead.

This kind of engineering effort makes sense at companies like Google and Meta because these companies spend such a large amount of money on computer resources to begin with.

Re: How Precision Time Protocol is being deployed at Meta

#17

Post by [deleted] »

throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

[deleted]

Re: How Precision Time Protocol is being deployed at Meta

#18

Post by foobiekr »

CapmCrackaWaka wrote: post #6

Why do so many tech companies seem to be releasing “secret sauce” for free lately? I see a lot of posts lately detailing how inner production systems work at large companies, and while I’m grateful, I’m curious why the higher ups think it’s worthwhile to release this information.

PTP isn’t secret sauce. Routers I worked on were doing PTP in like 2009. 1588 was standardized in 2002.

Re: How Precision Time Protocol is being deployed at Meta

#19

Post by jeffbee »

Great article, but I have to quibble about the casual ease with which they wave away the complexity of accessing these timestamps. The Linux APIs for accessing them is totally absurd. See for example the gRPC code that associates hardware (possibly) timestamps with messages, for tracing and other reasons. You have to re-arm the timestamp option before every send. And the whole concept of a timestamp for an ethernet frame maps poorly to stream sockets.

https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/...

Re: How Precision Time Protocol is being deployed at Meta

#20

Post by zh3 »

throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

It seems a little like it might mean that software errors caused by race conditions can be reduced by making the timing windows smaller. As this is a complex area (it might not appear so, but it is) the pragmatic solution could be to reduce the windows rather than fix the issue (maybe an FB engineer can speak OTR).
Post reply on HN