Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NTP at NIST Boulder Has Lost Power

lists.nanog.org

Post reply on HN
Print view View original

161–170 of 218 posts

Re: NTP at NIST Boulder Has Lost Power

#161

Post by toast0 »

axlee wrote: post #73
Animats wrote: post #13

Earlier quoted context omitted.

Google has their own fleet of atomic clocks and time servers. So does AWS. So does Microsoft. So does Ubuntu. They're not going to drift enough for months to cause trouble. So the Internet can ride through this, mostly. The main problem will be services that assume at least one of the NIST time servers is up. Somewhere, there's going to be something that won't work right when all the NIST NTP servers are down. But wh…

Can't they point these dns records to working servers meanwhile to avoid degradation?

If you use a general purpose hostname like time.nist.gov: that should resolve to an operational server and it makes sense to adjust during an incident. If you use a specific server hostname like time-a-b.nist.gov: that should resolve to the specific server and you're expected to have multiple hosts specified; it doesn't make sense to adjust during an incident, IMHO. You wanted boulder, you're getting boulder, faults and all.

Re: NTP at NIST Boulder Has Lost Power

#162

Post by mh- »

wpm wrote: post #110
Maxious wrote: post #41

Earlier quoted context omitted.

There's two other sites for the time.nist.gov service so it'll be okay. Probably more interesting is how you get a tier 0 site back in sync - NIST rents out these cyberpunk looking units you can use to get your local frequency standards up to scratch for ~$700/month https://www.nist.gov/programs-projects/frequency-measurement...

I must have one of those units oh my god

Someone needs to sell replicas (forgive the pun) of these.

Re: NTP at NIST Boulder Has Lost Power

#163

Post by CamperBob2 »

adastra22 wrote: post #69
misnome wrote: post #43

Earlier quoted context omitted.

All these use at least GPS for timing

No, they don’t. GPS is orders of magnitude less reliable than the most up to date metric time synchronization over fixed topology fiber links.

I doubt that very much. GPS time integrity is a big deal in many very important applications -- not the least of which is GPS itself -- and is treated as such.

Yes, an individual fiber distribution system can be much more accurate than GNSS time, but availability is what actually matters. Five nines at USNO would get somebody fired.

Re: NTP at NIST Boulder Has Lost Power

#165

Post by AlotOfReading »

misnome wrote: post #119
adastra22 wrote: post #69

Earlier quoted context omitted.

No, they don’t. GPS is orders of magnitude less reliable than the most up to date metric time synchronization over fixed topology fiber links.

I wonder why we bothered building GPS signal waveguides into the bottom of a mine then. Clearly we should have consulted the experts of hacker news first. Losing NTP for a day is going to affect fuck-all.

I'm not even sure why you're trying to argue this. It's well established that Time over Fiber is 1-2 orders of magnitude more accurate and precise than GNSS time. Fiber time is also immune to many of the numerous sources of interference GNSS systems encounter, which anyone who's done serious timekeeping will be well acquainted with.

Re: NTP at NIST Boulder Has Lost Power

#166

Post by CamperBob2 »

eichin wrote: post #99
makeitdouble wrote: post #32

Earlier quoted context omitted.

I think it wont be affected by this but on the top of my head: - GPS - industrial complex that synchronize operations (we could include trains) - telecoms in general (so a level higher than the internet)

GPS uses the atomic clocks on the satellites though. (Random search result from space force https://www.ssc.spaceforce.mil/Newsroom/Article/4039094/50-y... claims that cell phone tower-to-tower handoff uses GPS-mediated timing (only microsecond level though.)

The satellite clocks are designed to run autonomously for a few days without noticeable degradation, and up to a few weeks with defined levels of inaccuracy, but they are normally adjusted once a day by the ground stations based on the timescale maintained by the USNO. That, in turn, uses an ensemble of H-masers.

Re: NTP at NIST Boulder Has Lost Power

#167

Post by user3939382 »

It’s fine. The public pays for sci-fi clocks used by NIST and the Navy and we get shit latency over NTP and a WWVB signal that barely reaches a huge chunk of the country. CLOCKS WE PAID FOR. Jane Street gets lightning access to clocks and our pension managers get their NTP trades front run. NTP is a disgrace and an insult when it is working.

Re: NTP at NIST Boulder Has Lost Power

#168

Post by rgovostes »

glkindlmann wrote: post #24

Of the various internet .+P, NTP is one I never learned about as a student, so now I'm looking at its web page [1] by its creator David L. Mills (1938-2024). I've found one video of him giving a retrospective of his extensive internet work; he talks about NTP at 34:51 [2] and later at 56:26 [3]. [1] https://www.eecis.udel.edu/~mills/ntp.html [2] https://youtu.be/08jBmCvxkv4?si=WXJCV_v0qlZQK3m4&t=2092 [3] https://yout…

Amazingly, The New Yorker wrote an article about it: https://www.newyorker.com/tech/annals-of-technology/the-thor...

Re: NTP at NIST Boulder Has Lost Power

#169

Post by gorkish »

puzzlingcaptcha wrote: post #34
guenthert wrote: post #29

Earlier quoted context omitted.

Ubuntu using atomic clocks would surprise me. Sure they could, but it's not obvious to me why they would spend $$$$ on such. More plausible to me seems that they would be using GPSDO as reference clocks (in this context, about as good as your own atomic clock), iff they were running their own time servers. Google finds only that they are using servers from the NTP Pool Project, which will be using a variety of refere…

I think people have a wrong idea of what a modern atomic clock looks like. These are readily available commercially, Microchip for example will happily sell you hydrogen, cesium or rubidium atomic clocks. Hydrogen masers are rather unwieldy, but you can get a rubidium clock in a 1U format and cesium ones are not much bigger. I think their cesium freq standards are formerly a HP business they acquired. Example: https:…

woah hold on a sec. that's not how these clocks are actually used though.

It's a huge huge huge misconception that you can just plunk down an "atomic clock", discipline an NTP server with it and get perfect wallclock time out of it forever. That is just not how it works. Two hydrogen masers sitting next to each other will drift. Two globally distributed networks of hydrogen masers will drift. They cannot NOT drift. The universe just be that way.

UTC is by definition a consensus; there is no clock in the entire world that one could say is exactly tracking it.

Google probably has the gear and the global distribution that they could probably keep pretty close over 30-60 days, but they are assuredly not trying to keep their own independent time standard. Their goal is to keep events correlated on their own network, and for that they just need good internal distribution and consensus, and they are at the point where doing that internally makes sense. But this is the same problem on any size network.

Honestly for just NTP, I've never really seen evidence that anything better than a good GPS disciplined TCXO even matters. The reason they offer these oscillators in such devices is because they usually do additional duties like running PtP or distributing a local 10mhz reference where their specific performance characteristics are more useful. Rubidium, for instance, is very stable at short timescales but has awful long term stability.

Re: NTP at NIST Boulder Has Lost Power

#170

Post by Animats »

Youden wrote: post #76
genidoi wrote: post #17

Earlier quoted context omitted.

Atomic clock non-expert here, what does having a fleet of atomic clocks entail and why would the hyperscalers bother?

There's a lot of focus in this thread on the atomic clocks but in most datacenters, they're not actually that important and I'm dubious that the hyperscalers actually maintain a "fleet" of them, in the sense that there are hundreds or thousands of these clocks in their datacenters. The ultimate goal is usually to have a bunch of computers all around the world run synchronised to one clock, within some very small erro…

GPS satellites have their own atomic clocks. They're synchronized to clocks at the GPS control center at Schriever Space Force Base, Colorado, formerly Falcon AFB. They in turn synchronize to NIST in Boulder, Colorado. GPS has a lot of ground infrastructure checking on the satellites, and backup control centers. GPS should continue to work fine, even if there's some absolute error vs. NIST. Unless there have been layoffs.
Post reply on HN