Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FPGA NTP Server

research.wand.net.nz

Post reply on HN
Print view View original

41–50 of 70 posts

Re: FPGA NTP Server

#41

Post by gbrown_ »

lmilcin wrote: post #40
gbrown_ wrote: post #38

Earlier quoted context omitted.

Oh I wasn't meaning specifically in the context of NTP, just generally.

> Oh I wasn't meaning specifically in the context of NTP, just generally. So you decided to just answer with a generalization without regard for the particular case of NTP? Are you frequently building custom hardware when default OS settings do not suit your needs? Linux has a lot of configuration potential. It is wise to explore and understand various knobs available before you decide to complicate your life and bui…

Have you perhaps mixed up my comment with this one https://news.ycombinator.com/item?id=26152783?

I was just asking about your thoughts of how outside of trading jitter is seldom considered. I appreciate it's off-topic from the actual HN post but I was curious given your experience in trading.

Re: FPGA NTP Server

#42

Post by makomk »

wmf wrote: post #13
jbronn wrote: post #10

Earlier quoted context omitted.

I personally enjoyed the challenge of setting up PTP at home. Why would a hacker scoff at nanosecond-level timekeeping —- isn’t the entire internet a “telco/enterprise” thing?

To do PTP "right" requires every switch to support it and a NIC with hardware timestamps. Also, I've seen claims that PTP is no more precise than a good implementation of NTP.

The NIC with hardware timestamps part should be pretty easy if you're already implementing it using an FPGA like this project did - in fact, in a sense that seems to be exactly what they're doing with NTP. Finding switches that support it might be a little harder.

Re: FPGA NTP Server

#43

Post by willis936 »

sgtnoodle wrote: post #29
willis936 wrote: post #16

Earlier quoted context omitted.

This post didn’t sound right to me, but I realized that my raspi4 GPS NTP server has been running ntp and not chrony. Chrony is better at modeling non deterministic timing behavior, so I swapped to that. It’s been ten minutes now and chronyc tracking has been marching the offset down. It’s sub 1 us at this point. System time : 0.000000123 seconds fast of NTP time Last offset : +0.000000366 seconds How to get this pre…

Assuming you're using a PPS signal and a kernel driver, presumably there's an interrupt handler or perhaps a capture timer peripheral that is capturing a hardware timer when the PPS edge occurs. It doesn't matter too much when the userspace code gets around to adjusting the hardware timer as long as it can compute the difference between when the PPS edge came in and when it should have come in. The Linux API for fine…

Thank you for the detailed response! The PPS is currently driving a hardware interrupt on the raspberry pi that is read in by kernel mode software. My project is to drive an external display. Normally I would bypass the raspberry pi altogether and connect the PPS signal to the strobe input of the SIPO shift register. The problem is that the PPS signal cannot be trusted to always exist. Using a raspberry pi has a few benefits. Setting the timezone based on location, leap seconds, and smoothing out inconsistent GPS data. So while opting to use system time to drive the start of second adds error, I think the tradeoff for reliability is worth it.

I have considered adding complexity, such as adding a hardware mux to choose whether to use the GPS PPS signal or the raspberry pi's start-of-second. I should walk before I run though.

Re: FPGA NTP Server

#44

Post by Unklejoe »

lmilcin wrote: post #27

> Microprocessor based Network Time Protocol (NTP) servers suffer from a large amount of timestamp jitter, due to the hardware and Operating System (OS) being shared among other applications. So instead of configuring Linux correctly and dedicating a core for your NTP you have created custom hardware. Congratulations, now you have two problems. I have been working on algorithmic trading and it is not that hard to rel…

> I have been working on algorithmic trading and it is not that hard to reliably (like 100% of the time) respond within couple microseconds.

As someone who works in the flight test instrumentation industry (that primarily uses PTP rather than NTP), my first thought was "is a couple of microseconds supposed to be good?". With PTP, I usually achieve RMS offsets of I sort of agree with your point in that it's probably pointless implementing NTP inside of an FPGA, but I'd like to extend it further and say that if you really care about accurate timing, you should be using PTP rather than NTP anyway. In that case, an FPGA solution makes much more sense. However, in the systems I've worked with, the FPGA only implements a few pieces of core functionality - like the frame time stamping and the numerically-frequency-controlled clock. The actual protocol is usually implemented in software, except in the case of transparent switches.

That said, most modern NICs have PTP hardware, but you still usually need some external logic to actually use these synchronized clocks.

Re: FPGA NTP Server

#45

Post by Unklejoe »

lmilcin wrote: post #35
gbrown_ wrote: post #33

Earlier quoted context omitted.

> dedicating a core for your NTP Heh as soon as I read this I thought "algo trader?". If you don't mind me asking outside of trading how often do you see systems seriously manage jitter (specifically I'm thinking about interrupt management). I'm in HPC and whilst some such management is present it seems seldom thought about.

Heh, I already thought somebody will point it out. See, they decided to create custom hardware just to do NTP. How is instead dedicating a single core a worse solution? Setting dedicated core is basically configuration detail. There are no gains from custom hardware as network variability would mask them. If you want really good time guarantee because having right time is critical for your application, you put atomic…

> How is instead dedicating a single core a worse solution?

It is pretty much guaranteed to perform worse, if that's what you mean. Is it worth the effort and extra complexity? Probably not.

The timestamping jitter with an FPGA would probably be on the order of tens of nanoseconds while we're talking microseconds with a software solution.

> There are no gains from custom hardware as network variability would mask them.

I _think_ multiple cascaded jitters in a system add with the square root of the sum of squares, so there would still be an improvement.

Consider a system where you had 10 microseconds of inherent jitter, then you added another 10 microseconds on top of that. The total is not 10 microseconds - it would be 14.

Re: FPGA NTP Server

#46

Post by Unklejoe »

wmf wrote: post #13
jbronn wrote: post #10

Earlier quoted context omitted.

I personally enjoyed the challenge of setting up PTP at home. Why would a hacker scoff at nanosecond-level timekeeping —- isn’t the entire internet a “telco/enterprise” thing?

To do PTP "right" requires every switch to support it and a NIC with hardware timestamps. Also, I've seen claims that PTP is no more precise than a good implementation of NTP.

> To do PTP "right" requires every switch to support it and a NIC with hardware timestamps.

I agree, but PTP will in fact work over regular commercial switches on a LAN. The problem is that it will introduce jitter if there's other traffic on the network, but as long as the paths from master to slave (terms used in the standard) remain symmetric, you can filter this out and achieve performance almost as good as if you were using PTP transparent switches.

Re: FPGA NTP Server

#47

Post by LeonenTheDK »

magicalhippo wrote: post #26

For less demanding users, a Raspberry Pi and a cheap GPS module can do the trick[1]. I get less than +/-300us in error according to chronyc on the clients. Note that if you get one of the GPS modules, some do not expose the PPS pin to the headers, so might require some board modification (they use it just to drive a LED). I got one like this[2] that has it exposed. Also note that the small antennas only work outside,…

That reminded me of Mitxela's GPS clock[0]. Using GPS modules to get an incredibly accurate time is really interesting, I need to find some time to make one of these.

[0]https://mitxela.com/projects/precision_clock_mk_iii

Re: FPGA NTP Server

#48

Post by varjag »

mlichvar wrote: post #37
varjag wrote: post #31

Earlier quoted context omitted.

PTP at this point is ubiquitous and cheap. It is hard to source a PHY that has no support for it in 2021. > If they had the same support for NTP, it would perform as well. Unless you'd extend NTP to become PTP, it absolutely wouldn't.

Yes, NICs with support for hardware timestamping are common (it's typically in the MAC, not PHY), but switches that have a good support for PTP, either as a boundary clock, or transparent clock, are not cheap. At least I have not seen one yet. Do you have any examples? Some switches support NTP as a server and client (equivalent to the PTP boundary clock), but there don't seem to be any using hardware timestamping. I…

> Do you have any examples?

Probably all current Cisco offerings? Ubiquiti industrial switches? A whole crowd of second tier vendors like Lantech or Korenix? These are just those I had direct experience with.

Of these, Cisco definitely does boundary clock on L3, on at least several models of their routers.

> In my tests with directly connected NICs the synchronization is stable to few nanoseconds, same as with PTP.

Yes network sync is piece of cake if you drop the whole network bit. That said am slightly skeptical about ns level precision with NTP. Did you measure synchronicity between the two devices via scope?

Re: FPGA NTP Server

#49

Post by lmilcin »

Unklejoe wrote: post #44
lmilcin wrote: post #27

> Microprocessor based Network Time Protocol (NTP) servers suffer from a large amount of timestamp jitter, due to the hardware and Operating System (OS) being shared among other applications. So instead of configuring Linux correctly and dedicating a core for your NTP you have created custom hardware. Congratulations, now you have two problems. I have been working on algorithmic trading and it is not that hard to rel…

> I have been working on algorithmic trading and it is not that hard to reliably (like 100% of the time) respond within couple microseconds. As someone who works in the flight test instrumentation industry (that primarily uses PTP rather than NTP), my first thought was "is a couple of microseconds supposed to be good?". With PTP, I usually achieve RMS offsets of I sort of agree with your point in that it's probably p…

On a "normal" box the time is usually combination of various sources neither of which was really designed to provide accurate, sub microsecond, absolute time. (That except possibly for this "PTP hardware" in NICs that I don't know anything about)

So even if you somehow devise a piece of hardware to pass the accurate time to the operating system, there would be no way to track it accurately.

Now, I think that really accurate absolute timing on machines isn't really that critical. For example, for consistency protocols usually it is enough to provide guarantee of 1s around true time.

For algorithmic trading we were mostly interested in responding as quickly as possible to the event that came from the stock exchange. The packets always came with a known delay and the timing wasn't all that important except for debugging. Everything was done on a single box (so no need to synchronize with anything else) and even on that box the events were processed such that there was no need to coordinate with other threads, each thread just consumed, processed and published results.

Re: FPGA NTP Server

#50

Post by lmilcin »

Unklejoe wrote: post #45
lmilcin wrote: post #35

Earlier quoted context omitted.

Heh, I already thought somebody will point it out. See, they decided to create custom hardware just to do NTP. How is instead dedicating a single core a worse solution? Setting dedicated core is basically configuration detail. There are no gains from custom hardware as network variability would mask them. If you want really good time guarantee because having right time is critical for your application, you put atomic…

> How is instead dedicating a single core a worse solution? It is pretty much guaranteed to perform worse, if that's what you mean. Is it worth the effort and extra complexity? Probably not. The timestamping jitter with an FPGA would probably be on the order of tens of nanoseconds while we're talking microseconds with a software solution. > There are no gains from custom hardware as network variability would mask the…

> Is it worth the effort and extra complexity? Probably not.

That's pretty much definition of a worse solution.

Post reply on HN