Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a more precise time appliance

engineering.fb.com

Post reply on HN
Print view View original

31–40 of 188 posts

Re: Open-sourcing a more precise time appliance

#31

Post by jrockway »

What's interesting to me is how the form factor of timing GPS modules has stayed constant over the years. I started my GNSS timing journey with a used Trimble GPS from the 2000s, and it has the same form factor and pinout as the modern multi-constellation timing GPSes from uBlox.

I've had a GPS clock going for several years at this point, and without an atomic clock or really any fanciness (just LinuxPPS and Chrony), I see about +/- 380ns, which is pretty good. NTP to the Internet gives me jitter in the range of about 20ms-70ms, about 5 orders of magnitude worse.

(The version a few iterations ago looked like this: https://github.com/jrockway/beaglebone-gps-clock. But I now have a uBlox multi-constellation GPS, which is much more accurate with my limited view of the sky from my Brooklyn apartment. And I 3D printed the case, so it actually looks presentable instead of like some crazed madman that attacked a plastic case with a hacksaw -- which is exactly how I made the first case. As for the DS3231 RTC that I added... that seems to be stable within about 1.5us, which is pretty impressive. I tuned it a little bit with the trim register, though.)

My takeaway from this article is that the project seems quite compelling, but a lot of cost is added to support PTP. I see why Facebook needs that, but to sync time precisely throughout my lab, all I need is a coaxial cable with 10MHz on it. I don't mind if my workstation is 20ms off UTC.

Re: Open-sourcing a more precise time appliance

#33

Post by KaiserPro »

MisterTea wrote: post #22

> While building our Time Appliance, we also invented a Time Card, a PCIe card that can turn any commodity server into a time appliance. Invented? These cards have been around. e.g. Meinberg makes a variety of expansion cards which use various time keeping sources such as WWVB, IRIG-B, GNSS, etc.

I read it that they've invented a product called time card, rather than the concept.

Its an accurate and stable clock source. which is an important distinction. The Meinberg PCI cards appear not to have a local clock source thats well disciplined (ie a temperature compensated oscillator)

Having an accurate time source that is updated once every second (typically) with a 1pps GPS is grand for most things. But if you need a stable clock as well, then you want a tiny atomic clock onboard as well. This is so you can run a clock source for an entire datacenter, or local section.

Re: Open-sourcing a more precise time appliance

#34

Post by KaiserPro »

kd913 wrote: post #11
kfreds wrote: post #5

Earlier quoted context omitted.

From the article: "More accurate time keeping enables more advanced infrastructure management across our data centers, as well as faster performance of distributed databases."

That seems rather ambiguous and doesn't really explain why having more time precision results in faster performance or 'advanced infrastructure management'. I can understand why a stock broker or trading firm requires PTP to enable precise date-stamps for auditing/validating trades. I don't see how having a time granularity on the order of picoseconds is needed for a data center.

When you have collisions, the timestamp might be the thing that decides which update came first. The higher the precision of the time stamp, the faster you can run stuff.

Re: Open-sourcing a more precise time appliance

#35

Post by amelius »

MisterTea wrote: post #22

> While building our Time Appliance, we also invented a Time Card, a PCIe card that can turn any commodity server into a time appliance. Invented? These cards have been around. e.g. Meinberg makes a variety of expansion cards which use various time keeping sources such as WWVB, IRIG-B, GNSS, etc.

What sounds like the most reasonable strategy to me is to have a single atomic clock in every datacenter, and then some basic hardware in every server to communicate with it.

Re: Open-sourcing a more precise time appliance

#36

Post by throw0101a »

MisterTea wrote: post #28
michaelt wrote: post #18

A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their…

This gear will get cheaper soon. There is a big push in the Industrial automation world to get TSN/PTP deployed more widely and perhaps universally. The idea is to eliminate the dichotomy of real time vs non real time Ethernet networks in control systems from the plant network down to the machine level. This way you can send EtherCAT frames at equidistant intervals to servo drives in the microsecond range while casua…

> single pair Ethernet

For anyone wondering:

> In addition to the more computer-oriented two and four-pair variants, the 100BASE-T1 and 1000BASE-T1 single-pair Ethernet PHYs are intended for automotive applications[17] or as optional data channels in other interconnect applications.[18] The single pair operates at full duplex and has a maximum reach of 15 m or 49 ft (100BASE-T1, 1000BASE-T1 link segment type A) or up to 40 m or 130 ft (1000BASE-T1 link segment type B) with up to four in-line connectors. Both PHYs require a balanced twisted pair with an impedance of 100 Ω. The cable must be capable of transmitting 600 MHz for 1000BASE-T1 and 66 MHz for 100BASE-T1.

> Similar to PoE, Power over Data Lines (PoDL) can provide up to 50 W to a device.[19]

* https://en.wikipedia.org/wiki/Ethernet_over_twisted_pair#Sin...

> The IEEE 802.3bu-2016[12] amendment introduced single-pair Power over Data Lines (PoDL) for the single-pair Ethernet standards 100BASE-T1 and 1000BASE-T1 intended for automotive and industrial applications.[13] On the two-pair or four-pair standards, power is transmitted only between pairs, so that within each pair there is no voltage present other than that representing the transmitted data. With single-pair Ethernet, power is transmitted in parallel to the data. PoDL defines 10 power classes, ranging from .5 to 50 W (at PD).

* https://en.wikipedia.org/wiki/Power_over_Ethernet#PoDL

100BASE-T1 is IEEE 802.3bw-2015, 1000BASE-T1 is IEEE 802.3bp-2016. 2.5 Gb/s, 5 Gb/s, and 10 Gb/s over a single pair is 802.3ch-2020: the focus of these is in the embedded automotive space.

* https://blog.siemon.com/standards/ieee-p802-3ch-multi-gig-au...

Re: Open-sourcing a more precise time appliance

#37

Post by myself248 »

amelius wrote: post #35
MisterTea wrote: post #22

> While building our Time Appliance, we also invented a Time Card, a PCIe card that can turn any commodity server into a time appliance. Invented? These cards have been around. e.g. Meinberg makes a variety of expansion cards which use various time keeping sources such as WWVB, IRIG-B, GNSS, etc.

What sounds like the most reasonable strategy to me is to have a single atomic clock in every datacenter, and then some basic hardware in every server to communicate with it.

Which is exactly how the telephone industry has done it for decades, it's called a "building-integrated timing supply", or BITS clock. Typically this is a GPS-referenced clock with a local quartz or rubidium holdover oscillator, and output cards with dozens or hundreds of outputs serving various machines in the office.

Re: Open-sourcing a more precise time appliance

#38

Post by delayclose2 »

> The offset is practically 0 ranging within nanoseconds range.

Uhm, no, the graph clearly shows that the offset is varying between 0-2 microseconds, which is 0 - 2,000 nanoseconds.

2,000 nanoseconds is not "practically zero". PTP precision is supposed to be in the 10ns range.

Also, how did they get 40us precision on (what I assume to be) a normal kernel on a normal x86 box? I would have assumed that in a datacenter environment, with traffic levels being random, the jitter introduced in the kernel networking stack alone would be in the hundreds of microseconds at least.

Re: Open-sourcing a more precise time appliance

#39

Post by michaelt »

rkangel wrote: post #4

I'm still unclear why they need the MAC on top of the GNSS reception. I can understand if it's just to provide holdover in event of GNSS signal loss, but they seem to be saying that it provides better precision when you've got both.

Although clock holdover might seem like a pointless feature - GPS outages are very unusual* - what holdover is mostly for is giving you time to respond if someone clumsily knocks your antenna over, or crushes your coax cable, or it gets damage that makes it fill with water, or something like that.

A lot of the demand for high-precision clocks is for cell phone base stations, where there's no guarantee there'll be someone on hand to make repairs promptly.

* They happen occasionally, of course.

Re: Open-sourcing a more precise time appliance

#40

Post by nonameiguess »

> Now, as long as printing the PCB and soldering tiny components does not sound scary, anyone can build their own Time Card for a fraction of the cost of a regular time appliance.

Doesn't sound scary, but not a lot of research labs and data centers are going to want to build their expansion cards from parts, even given free access to instructions. I understand if there isn't enough of a market for this to make it a viable commercial product, but when you have more money than God anyway, what's the reason not to make a few extra and donate or sell them to the small number of users who can use it?

Post reply on HN