Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FPGA NTP Server

research.wand.net.nz

Post reply on HN
Print view View original

51–60 of 70 posts

Re: FPGA NTP Server

#51

Post by Unklejoe »

lmilcin wrote: post #50
Unklejoe wrote: post #45

Earlier quoted context omitted.

> How is instead dedicating a single core a worse solution? It is pretty much guaranteed to perform worse, if that's what you mean. Is it worth the effort and extra complexity? Probably not. The timestamping jitter with an FPGA would probably be on the order of tens of nanoseconds while we're talking microseconds with a software solution. > There are no gains from custom hardware as network variability would mask the…

> Is it worth the effort and extra complexity? Probably not. That's pretty much definition of a worse solution.

Not necessarily when it comes to something like HFT, where they do things like run point to point microwave links to reduce latency by microseconds.

Re: FPGA NTP Server

#52

Post by Bootvis »

hansor wrote: post #36
gbrown_ wrote: post #33

Earlier quoted context omitted.

> dedicating a core for your NTP Heh as soon as I read this I thought "algo trader?". If you don't mind me asking outside of trading how often do you see systems seriously manage jitter (specifically I'm thinking about interrupt management). I'm in HPC and whilst some such management is present it seems seldom thought about.

NTP is hilariously insufficient for HFT, we have far better protocol around: PTP. https://en.wikipedia.org/wiki/Precision_Time_Protocol

Jane Street has an interesting podcast on this very subject: https://signalsandthreads.com/clock-synchronization/

Re: FPGA NTP Server

#54

Post by 1_person »

lmilcin wrote: post #27

> Microprocessor based Network Time Protocol (NTP) servers suffer from a large amount of timestamp jitter, due to the hardware and Operating System (OS) being shared among other applications. So instead of configuring Linux correctly and dedicating a core for your NTP you have created custom hardware. Congratulations, now you have two problems. I have been working on algorithmic trading and it is not that hard to rel…

The uncontrollable and unobservable SMM interrupts of most modern CPUs add sufficient jitter to the reference clock sampling that there is no correct configuration.

You generally must use an FPGA or appropriate DSP/microcontroller to achieve precision beyond ~10s of nanoseconds, which is entirely achievable even with NTP between hosts that have low contention 10+Gbps per transceiver lane interfaces / properly configured DCB/QoS so the time synchronization packets always egress without delay. PTP can, of course, achieve even better precision.

Round trip time / transmission time ("respond within couple microseconds") are irrelevant, it's a rising or falling edge feature of the packet burst which timestamps the sample and the interval between these samples that's used to train the clocks. This can be accurate to within femtoseconds at the extreme.

Re: FPGA NTP Server

#55

Post by 1_person »

Unklejoe wrote: post #45
lmilcin wrote: post #35

Earlier quoted context omitted.

Heh, I already thought somebody will point it out. See, they decided to create custom hardware just to do NTP. How is instead dedicating a single core a worse solution? Setting dedicated core is basically configuration detail. There are no gains from custom hardware as network variability would mask them. If you want really good time guarantee because having right time is critical for your application, you put atomic…

> How is instead dedicating a single core a worse solution? It is pretty much guaranteed to perform worse, if that's what you mean. Is it worth the effort and extra complexity? Probably not. The timestamping jitter with an FPGA would probably be on the order of tens of nanoseconds while we're talking microseconds with a software solution. > There are no gains from custom hardware as network variability would mask the…

The precision achievable with FPGAs is well beyond 10s of nanoseconds.

The high rate transceivers are used with capacitative dividers or internal signal propagation delays defining the reference intervals for time to digital conversion in extreme accuracy applications, these and similar approaches yield precision to picosecond scale and beyond.

Re: FPGA NTP Server

#56

Post by lmilcin »

1_person wrote: post #54
lmilcin wrote: post #27

> Microprocessor based Network Time Protocol (NTP) servers suffer from a large amount of timestamp jitter, due to the hardware and Operating System (OS) being shared among other applications. So instead of configuring Linux correctly and dedicating a core for your NTP you have created custom hardware. Congratulations, now you have two problems. I have been working on algorithmic trading and it is not that hard to rel…

The uncontrollable and unobservable SMM interrupts of most modern CPUs add sufficient jitter to the reference clock sampling that there is no correct configuration. You generally must use an FPGA or appropriate DSP/microcontroller to achieve precision beyond ~10s of nanoseconds, which is entirely achievable even with NTP between hosts that have low contention 10+Gbps per transceiver lane interfaces / properly configu…

Let's say you get FPGA and then what? Where are you going to use that signal? On a server that has "uncontrollable and unobservable SMM interrupts"?

Re: FPGA NTP Server

#57

Post by 1_person »

lmilcin wrote: post #56
1_person wrote: post #54

Earlier quoted context omitted.

The uncontrollable and unobservable SMM interrupts of most modern CPUs add sufficient jitter to the reference clock sampling that there is no correct configuration. You generally must use an FPGA or appropriate DSP/microcontroller to achieve precision beyond ~10s of nanoseconds, which is entirely achievable even with NTP between hosts that have low contention 10+Gbps per transceiver lane interfaces / properly configu…

Let's say you get FPGA and then what? Where are you going to use that signal? On a server that has "uncontrollable and unobservable SMM interrupts"?

Sure, why not?

It's going to offer an incremental improvement in accuracy for the server's derived clock when the reference clock is not affected by that and other sources of jitter.

Beyond the accuracy, the hardware is significantly cheaper and more efficient than a server with a general purpose x86 CPU.

For a dedicated network reference clock, an FPGA or ASIC solution is simply better in every measurable way.

It is more complex, to be sure, but the complexity needn't be your concern.

Re: FPGA NTP Server

#58

Post by traceroute66 »

Boy is this a timely thread !

I've been looking into doing a little COVID lockdown project consisting of a multi-GNSS timeserver.

Does anyone have examples of projects using something like the UBlox ZED-F9T [1] which can do concurrent connections to 4 GNSS constellations ?

I'm not an electrical engineer, so unfortunatley I lack the skills to design a PCB from scratch. ;-(

[1]https://www.u-blox.com/en/product/zed-f9t-module

Re: FPGA NTP Server

#59

Post by Unklejoe »

1_person wrote: post #55
Unklejoe wrote: post #45

Earlier quoted context omitted.

> How is instead dedicating a single core a worse solution? It is pretty much guaranteed to perform worse, if that's what you mean. Is it worth the effort and extra complexity? Probably not. The timestamping jitter with an FPGA would probably be on the order of tens of nanoseconds while we're talking microseconds with a software solution. > There are no gains from custom hardware as network variability would mask the…

The precision achievable with FPGAs is well beyond 10s of nanoseconds. The high rate transceivers are used with capacitative dividers or internal signal propagation delays defining the reference intervals for time to digital conversion in extreme accuracy applications, these and similar approaches yield precision to picosecond scale and beyond.

Right, but Ethernet timestamping usually occurs on a symbol or byte level inside the MAC, so for GigE, you're looking at 8 nanoseconds per symbol. This could be improved by oversampling the symbols, but at least with GigE, there's (I think) 4 cycles of uncertainty each time the link comes up due to how the clocking agreement between master/slave is established. It's also somewhat difficult to run large counters faster than a few hundred MHz inside of an FPGA. I've worked with systems that use two different clocks - one small counter for Ethernet timestamping, and another main clock, with a periodic latching between the two to correlate them, but it's uncommon.

There's also the fact that most PHYs operate by having a FIFO with data being clocked in and clocked out on different clock domains. The way I understand it, the PHY FIFO fills up half way, and only then does it start draining into the MAC. This is to allow for oscillator frequency errors between each device. This is also why there's a maximum frame size and max oscillator tolerance specified for Ethernet - so that this FIFO doesn't under or overrun before the entire frame is received.

I think sync-e can improve this quite a bit.

Of course, FPGAs themselves are capable of much tighter timings, but in practice, it's usually implemented this way.

Re: FPGA NTP Server

#60

Post by sgtnoodle »

willis936 wrote: post #43
sgtnoodle wrote: post #29

Earlier quoted context omitted.

Assuming you're using a PPS signal and a kernel driver, presumably there's an interrupt handler or perhaps a capture timer peripheral that is capturing a hardware timer when the PPS edge occurs. It doesn't matter too much when the userspace code gets around to adjusting the hardware timer as long as it can compute the difference between when the PPS edge came in and when it should have come in. The Linux API for fine…

Thank you for the detailed response! The PPS is currently driving a hardware interrupt on the raspberry pi that is read in by kernel mode software. My project is to drive an external display. Normally I would bypass the raspberry pi altogether and connect the PPS signal to the strobe input of the SIPO shift register. The problem is that the PPS signal cannot be trusted to always exist. Using a raspberry pi has a few…

If you want to precisely generate a PPS edge in software with less jitter than you can schedule, you can use a PWM peripheral. Wake up a few milliseconds before the PPS edge is due, get the system time, and compute the precise time until the PPS is due. Initialize the PWM peripheral to transition that far into the future, then go back to sleep until a bit after the transition should have happened, and disable the PWM peripheral.

This works because a thread of execution generally knows what time it is with higher precision than it can accurately schedule itself.

I'm not sure I understand how you're using a PPS signal to drive a display, though. Is it an LED segment display? I assume you want it to update once a second, precisely on the edge of each second. Displays generally exist for humans, though, and a human isn't going to perceive a few milliseconds of jitter on a 1Hz update.

Post reply on HN