Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Precision Time Protocol is being deployed at Meta

engineering.fb.com

Post reply on HN
Print view View original

91–100 of 104 posts

Re: How Precision Time Protocol is being deployed at Meta

#91

Post by willis936 »

zie wrote: post #58
willis936 wrote: post #26

Earlier quoted context omitted.

Just because high precision tasks don't care about time of day as much, PTP still uses TAI timestamps. The only time of day information missing is leap seconds, and those were a mistake that is being corrected. You don't need NTP for time of day when PTP is used.

The point being you need a source of truth for PTP to figure out what the current date and time are. You won't be pushing PTP over the public internet, that's ridiculous. You can use a GPS/sat sync or atomic clock or NTP, doesn't matter, but something has to tell PTP what time it is. If you don't, then you will be subject to quartz crystal drift. It may or may not matter to your application(s), but for most people th…

Is that really useful though? As soon as you need to synchronize your dataset beyond the local PTP network NTP-based timestamping becomes useless. You need a local GPS-based time reference backing PTP.

Re: How Precision Time Protocol is being deployed at Meta

#92

Post by bradknowles »

SEJeff wrote: post #77
throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

Disclaimer: I work in finance, and have for 15+ years. > Given the special hardware needed for the grand master clock to get down to nanosecond time scales, I'm doubtful this will be used in most data centres of most corporate networks. The "special hardware" is often just a gps antenna and a PCI card though. In fact, many tier 1 datacenters actually provide a "service" where they'll either cross connect you directly…

Disclaimer: I've been involved in supporting the NTP Public Services Project since 2003.

I assure you, with the right hardware and paying attention to your latencies, NTP can get you down below one millisecond accuracy. Poul Henning Kamp was doing nanosecond level accuracy with NTP back in the mid-aughts, but then he had rewritten the NTP server code, the NTP client code, and the kernel on the server.

As an NTP service provider, what you really want to keep an eye on is the Clock Error Bound that gives you the worst case estimate for how bad the time is that you could be serving to your customers. For the client side, you mainly care about just the accuracy you're actually getting.

Re: How Precision Time Protocol is being deployed at Meta

#93

Post by bradknowles »

pclmulqdq wrote: post #50
throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

I have never seen NTP in a datacenter of reasonable size get below an error of 1e-4. PTP without custom hardware can easily get 3 orders of magnitude better.

PTP is harder to do within a datacenter. You either need hardware support in every switch and network interface, or you're doing software timestamping at which point you might as well be using NTP. And PTP doesn't support DNS, only IPv4 or IPv6 or Layer 2 MAC addresses.

PTP also requires some intelligence with regards to configuring your Ordinary Clocks, your Transparent Clocks, and your Border Clocks. And you have to have these configured on every device in the network path.

PTP does have a unicast mode as well as multicast, which can help eliminate unknowable one-way latencies.

It's a pain.

Check the documentation at https://linuxptp.nwtime.org/documentation/ and especially https://linuxptp.nwtime.org/documentation/ptp4l/

Re: How Precision Time Protocol is being deployed at Meta

#94

Post by bradknowles »

stingraycharles wrote: post #80
throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

> But given that NTP (either ntpd or chrony) tends to give me an estimated error of around (tens of) 1e-6 seconds, and PTP can get down to 1e-9 seconds, I'm not sure how many data centre applications need that level of accuracy. I know that in trading, auditing trades / order books requires extremely accurate timing, and they typically deploy GPS hardware to get the required level of accuracy. As GPS is accurate to a…

A good Stratum-1 GNSS from a company like Meinberg or Microchip will include a Rubidium or Cesium reference clock that is then disciplined by GPS and can get you down to sub-nanosecond level accuracy.

Re: How Precision Time Protocol is being deployed at Meta

#95

Post by bradknowles »

throw0101a wrote: post #86
stingraycharles wrote: post #80

Earlier quoted context omitted.

> But given that NTP (either ntpd or chrony) tends to give me an estimated error of around (tens of) 1e-6 seconds, and PTP can get down to 1e-9 seconds, I'm not sure how many data centre applications need that level of accuracy. I know that in trading, auditing trades / order books requires extremely accurate timing, and they typically deploy GPS hardware to get the required level of accuracy. As GPS is accurate to a…

> […] improvement needed to not need GPS hardware anymore. You're simply trading NTP hardware for PTP hardware (grandmaster clocks). There is no way to get to 1e-9 scales without hardware support.

These days, all the vendors I know of are shipping hardware that does both. So, it's not just an NTP server, it's also a PTP server. Maybe you don't use one or the other part of that functionality, or maybe they are licensed separately, but they are there.

Re: How Precision Time Protocol is being deployed at Meta

#96

Post by bradknowles »

DannyBee wrote: post #66
throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

First, 300mm is not the real measure in practice for the common use case. PTP is often used to distribute GPS time for things that need it but don't have direct satellite access, and also so you don't have to have direct satellite access everywhere. For that use case, 1ns of inaccuracy is about 10ft all told (IE accounting for all the inaccuracy it generates). It can be less these days, especially if not just literal…

That's the one I was thinking about getting for my home lab. I'm also looking at: https://www.meinbergglobal.com/english/products/synchronizat...

I already have an ancient Meinberg Stratum-1 somewhere that I should pull out of storage and send back to Heiko so that they can put it in a museum. These days, for proper datacenter use, I'd go for something like this one: https://www.meinbergglobal.com/english/products/modular-2u-s...

Re: How Precision Time Protocol is being deployed at Meta

#97

Post by bradknowles »

willis936 wrote: post #26
kazinator wrote: post #3

It is not accurate to say that PTP is a predecessor to NTP. You need both. PTP synchronizes clocks to a ridiculous precision, like down to nanoseconds. To do that, it uses support in the ethernet hardware. Hardware adds precision stamps to the PTP-related ethernet frames, so that the time calculations are free of jitters induced by the higher network stacks. A cool thing is that PTP-aware routers/switches can actuall…

Just because high precision tasks don't care about time of day as much, PTP still uses TAI timestamps. The only time of day information missing is leap seconds, and those were a mistake that is being corrected. You don't need NTP for time of day when PTP is used.

PTP uses the time stamps that are configured. By default it might use TAI, but that's not the only option.

Feel free to look at the documentation I linked above.

Re: How Precision Time Protocol is being deployed at Meta

#98

Post by bradknowles »

willis936 wrote: post #91
zie wrote: post #58

Earlier quoted context omitted.

The point being you need a source of truth for PTP to figure out what the current date and time are. You won't be pushing PTP over the public internet, that's ridiculous. You can use a GPS/sat sync or atomic clock or NTP, doesn't matter, but something has to tell PTP what time it is. If you don't, then you will be subject to quartz crystal drift. It may or may not matter to your application(s), but for most people th…

Is that really useful though? As soon as you need to synchronize your dataset beyond the local PTP network NTP-based timestamping becomes useless. You need a local GPS-based time reference backing PTP.

Most clients won't speak PTP or know how to deal with it.

But NTP is something that many can deal with.

So, use PTP as a refclock for your NTP service, and then let the clients deal with an interface they already know and understand.

Those clients that really care can deal with the additional complexities of PTP, if you want to let them.

Re: How Precision Time Protocol is being deployed at Meta

#99

Post by bradknowles »

epgui wrote: post #2

This is incredible. If I knew I'd be working on things like this (as opposed to optimizing ad revenue or "user engagement" by A/B testing button copy), I'd possibly consider working at Meta.

We're doing some interesting things with time sync at AWS, and we are doing so at scale for millions of customers around the world, unlike the small scale stuff that Meta has announced here. Most recently, our team recently announced the Amazon NTP Pools service at time.aws.com.

We've got more cool stuff we are working on.

If we can get past this hiring freeze problem, we still have a couple of slots we need to fill. If you know a lot about time sync and the NTP and PTP protocols, and you have a passion for building some of the biggest impact systems on the planet, let me know.

Re: How Precision Time Protocol is being deployed at Meta

#100

Post by bradknowles »

error503 wrote: post #71
amluto wrote: post #59

I’m wondering whether the extremely careful GNSS part is really needed. A microsecond of offset between two servers in the same datacenter could easily matter, but I suspect that, if an entire datacenter were off by a microsecond, everything would be fine — communicating from that datacenter to anywhere else will take well over a microsecond, so an offset of this type would be a bit like the datacenter wiggling aroun…

Accurate delay compensation is necessary to enable redundancy. IF you need multiple GrandMasters at different locations in the facility, using independent RFoF systems and antennas, they will have different GNSS delays, and that difference will propagate down into uncertainty on the hosts. There are other ways to eliminate this uncertainty than to characterize the full delay, but if the racks are on opposite ends of…

It's a hard problem to solve. You end up doing something like the NIST TMAS service (see https://www.nist.gov/programs-projects/time-measurement-and-...) using differential common view measurements to create a "Multi-Source Common-View Disciplined Clock".
Post reply on HN