How Precision Time Protocol is being deployed at Meta • Hacker News Board

Re: How Precision Time Protocol is being deployed at Meta

#101

Post by bradknowles » Wed, Nov 23, 2022, 7:26 AM UTC

amluto wrote: post #70

walrus01 wrote: post #60

Earlier quoted context omitted.

Having rooftop mounted GNSS receive antennas for GPS+GLONASS is extremely common in telecom and ISP infrastructure applications. It's sort of a belt and suspenders approach to obtaining time from low stratum NTP sources and also having a local GNSS timing source to reference from. Or for use in a case where the network has a total absence of connectivity to any internet-based NTP sources (maybe because your managemen…

Meta seems to have gone with the expensive option. I think it’s this: https://www.mouser.com/ProductDetail/HUBER%2bSUHNER/Direct-G... Admittedly, at their scale, this is peanuts. But I wouldn’t buy one of these for a scrappy startup :). SparkFun will sell a perfectly serviceable kit for a few hundred dollars. (If you are worried about lightning, then GPSoF looks like cheap insurance.)

Their Calnex Sentinel equipment they use for measuring their signal is probably a lot more expensive than that (see https://calnexsol.com/en/product-detail/1033-sentinel).

You wouldn't want to build your entire monitoring system on top of that and be forced to deploy those at hundreds of datacenters around the world.

Re: How Precision Time Protocol is being deployed at Meta

#102

Post by throw0101a » Wed, Nov 23, 2022, 1:44 PM UTC

SEJeff wrote: post #77

throw0101a wrote: post #7

So they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > Th…

Disclaimer: I work in finance, and have for 15+ years. > Given the special hardware needed for the grand master clock to get down to nanosecond time scales, I'm doubtful this will be used in most data centres of most corporate networks. The "special hardware" is often just a gps antenna and a PCI card though. In fact, many tier 1 datacenters actually provide a "service" where they'll either cross connect you directly…

> However, as a protocol, NTP only guarantees 1 second accuracy.

The "one second" number is not inherent to the protocol, but comes from a survey from 1999:

> A recent survey[2] suggests that 90% of the NTP servers have network delays below 100ms, and about 99% are synchronized within one second to the synchronization peer.

* http://www.ntp.org/ntpfaq/NTP-s-algo.htm#Q-ACCURATE-CLOCK

A 2005 survey found (AFAICT, see Figure 1) that north of 99% — more like 99.5% — of servers had offsets less than 100ms, and that 90% have offsets less than 10ms:

* PDF: https://web.archive.org/web/20081221080840/http://www.ntpsur...

The output of "chronyc tracking" from a randomly-selected system I help run:

    Reference ID    : […]
    Stratum         : 3
    Ref time (UTC)  : Wed Nov 23 13:35:21 2022
    System time     : 0.000002993 seconds fast of NTP time
    Last offset     : +0.000003275 seconds
    RMS offset      : 0.000008091 seconds
    Frequency       : 13.191 ppm slow
    Residual freq   : +0.001 ppm
    Skew            : 0.017 ppm
    Root delay      : 0.001615164 seconds
    Root dispersion : 0.000048552 seconds
    Update interval : 65.3 seconds
    Leap status     : Normal

So chrony's algorithm thinks that the system time is currently off by 0.000 002 993 seconds (3e-6), and on average it is off by 0.000 008 091 seconds (8e-6).

All I had to do to achieve this was have some infra nodes (that do other things as well) point to *.pool.ntp.org, and then have my cattle and pets point their NTP software to those infra nodes. No special hardware, no special software, no configuring of network switches and routers: just a few lines of Ansible and I get (estimated) microsecond (1e-6) error levels.

Re: How Precision Time Protocol is being deployed at Meta

#103

Post by SEJeff » Wed, Nov 23, 2022, 3:32 PM UTC

bradknowles wrote: post #92

SEJeff wrote: post #77

Earlier quoted context omitted.

Disclaimer: I work in finance, and have for 15+ years. > Given the special hardware needed for the grand master clock to get down to nanosecond time scales, I'm doubtful this will be used in most data centres of most corporate networks. The "special hardware" is often just a gps antenna and a PCI card though. In fact, many tier 1 datacenters actually provide a "service" where they'll either cross connect you directly…

Disclaimer: I've been involved in supporting the NTP Public Services Project since 2003. I assure you, with the right hardware and paying attention to your latencies, NTP can get you down below one millisecond accuracy. Poul Henning Kamp was doing nanosecond level accuracy with NTP back in the mid-aughts, but then he had rewritten the NTP server code, the NTP client code, and the kernel on the server. As an NTP servi…

Yes, I've seen it get down to a few milliseconds of sync on the right hardware (boundary clock on the switches, stratum 0 timeserver with pps, etc), but the protocol only guarantees 1 second of sync. Am I incorrect in that assertion?

Re: How Precision Time Protocol is being deployed at Meta

#104

Post by zie » Wed, Nov 23, 2022, 4:06 PM UTC

willis936 wrote: post #91

zie wrote: post #58

Earlier quoted context omitted.

The point being you need a source of truth for PTP to figure out what the current date and time are. You won't be pushing PTP over the public internet, that's ridiculous. You can use a GPS/sat sync or atomic clock or NTP, doesn't matter, but something has to tell PTP what time it is. If you don't, then you will be subject to quartz crystal drift. It may or may not matter to your application(s), but for most people th…

Is that really useful though? As soon as you need to synchronize your dataset beyond the local PTP network NTP-based timestamping becomes useless. You need a local GPS-based time reference backing PTP.

It totally depends on your use case, and what you actually need in reality.

In general, once you need/want anything past what NTP can deliver for you, you really, really need to understand your use case and then design your time services to meet your specific needs.