Live data from Hacker News

NTP at NIST Boulder Has Lost Power

lists.nanog.org

31–40 of 218 posts

Re: NTP at NIST Boulder Has Lost Power

#31
post #10
post #5

Can anybody expand on the implications of this? Being unfamiliar with it, it's hard to tell if this is a minor blip that happens all the time, or if it's potentially a major issue that could cause cascading errors equal to the hype of Y2K.

Time travel is extremely dangerous right now. I highly recommend deferring time travel plans except for extreme temporal emergencies.

Define “extreme”?

Re: NTP at NIST Boulder Has Lost Power

#32

Earlier quoted context omitted.

I know this is HN, but the internet is pretty low on the list of things NIST time standards are important for.

could you list 3 things that you think are more important than the internet? (I know the internet is going to be fine; I just want to understand what you think ranks higher globally...)

I think it wont be affected by this but on the top of my head:

- GPS

- industrial complex that synchronize operations (we could include trains)

- telecoms in general (so a level higher than the internet)

Re: NTP at NIST Boulder Has Lost Power

#33
post #14

Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.

Somewhat interesting that they themselves don't have access to the site. You'd think there would have been some disaster plans put in place?

Re: NTP at NIST Boulder Has Lost Power

#34
post #13

Earlier quoted context omitted.

Google has their own fleet of atomic clocks and time servers. So does AWS. So does Microsoft. So does Ubuntu. They're not going to drift enough for months to cause trouble. So the Internet can ride through this, mostly. The main problem will be services that assume at least one of the NIST time servers is up. Somewhere, there's going to be something that won't work right when all the NIST NTP servers are down. But wh…

Ubuntu using atomic clocks would surprise me. Sure they could, but it's not obvious to me why they would spend $$$$ on such. More plausible to me seems that they would be using GPSDO as reference clocks (in this context, about as good as your own atomic clock), iff they were running their own time servers. Google finds only that they are using servers from the NTP Pool Project, which will be using a variety of refere…

I think people have a wrong idea of what a modern atomic clock looks like. These are readily available commercially, Microchip for example will happily sell you hydrogen, cesium or rubidium atomic clocks. Hydrogen masers are rather unwieldy, but you can get a rubidium clock in a 1U format and cesium ones are not much bigger. I think their cesium freq standards are formerly a HP business they acquired.

Example: https://www.microchip.com/en-us/products/clock-and-timing/co...

Re: NTP at NIST Boulder Has Lost Power

#35
post #30
post #17

Earlier quoted context omitted.

Atomic clock non-expert here, what does having a fleet of atomic clocks entail and why would the hyperscalers bother?

Spanner depends on having a time source with bounded error to maintain consistency. Google accomplishes this by having GPS and atomic clocks in several datacenters. https://static.googleusercontent.com/media/research.google.c... https://static.googleusercontent.com/media/research.google.c...

And more importantly, the tighter the time bound, the higher the performance, so more accurate clocks easily pay for themselves in other saved infrastructure costs to service the same number of users.

Re: NTP at NIST Boulder Has Lost Power

#36
post #12

This was an NTP 0 server right? What is the actual failback mechanism when that level of NTP server fails? This is some level of eldritch magic that I am aware of, but not familiar with but am interested in learning.

There are lots of Stratum 0 servers out there; basically anything with an atomic clock will do. They all count seconds independently from one another, all slowly diverging over time, with offset intervals being measured by mutual synchronization using a number of means (how is this done is interesting all by itself). Some atomic clocks are more accurate than others, and an ensemble of these is typically regarded as 'the' master clock.

To quote the ITU: "UTC is based on about 450 atomic clocks, which are maintained in 85 national time laboratories around the world." https://www.itu.int/hub/2023/07/coordinated-universal-time-a...

Beyond this, as other commenters have said, anyone who is really dependent on having exact time (such as telcos, broadcasters, and those running global synchronized databases) should have their own atomic clock fleets. There are thousands and thousands of atomic clocks in these fleets worldwide. Moreover, GPS time, used by many to act as their time reference, is distributed by yet other means.

Nothing bad will happen, except to those who have deliberately made these specific Stratum 0 clocks their only reference time. Anyone who has either left their computer at its factory settings or has set up their NTP configuration in accordance to recommended settings will be unaffected by this.

Re: NTP at NIST Boulder Has Lost Power

#37
post #33
post #14

Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.

Somewhat interesting that they themselves don't have access to the site. You'd think there would have been some disaster plans put in place?

Step One of most disaster plans is not to create a second emergency.

Re: NTP at NIST Boulder Has Lost Power

#38
post #10
post #5

Can anybody expand on the implications of this? Being unfamiliar with it, it's hard to tell if this is a minor blip that happens all the time, or if it's potentially a major issue that could cause cascading errors equal to the hype of Y2K.

Time travel is extremely dangerous right now. I highly recommend deferring time travel plans except for extreme temporal emergencies.

Same for database transaction roll back and roll forward actions.

And most enterprises, including banks, use databases.

So by bad luck, you may get a couple of transactions reversed in order of time, such as a $20 debit incorrectly happening before a $10 credit, when your bank balance was only $10 prior to both those transactions. So your balance temporarily goes negative.

Now imagine if all those amounts were ten thousand times higher ...

Re: NTP at NIST Boulder Has Lost Power

#39

Of the various internet .+P, NTP is one I never learned about as a student, so now I'm looking at its web page [1] by its creator David L. Mills (1938-2024). I've found one video of him giving a retrospective of his extensive internet work; he talks about NTP at 34:51 [2] and later at 56:26 [3]. [1] https://www.eecis.udel.edu/~mills/ntp.html [2] https://youtu.be/08jBmCvxkv4?si=WXJCV_v0qlZQK3m4&t=2092 [3] https://yout…

In [3] he mentions that one can use NTP to observe frequency deviations and use it as an early warning system for fire and AC failure. That really intrigues me. Can you actually? Has this ever been implemented?

Re: NTP at NIST Boulder Has Lost Power

#40

Earlier quoted context omitted.

But pretty high on the list that NIST NTP is important for (since it leaves the building through the internet).

If NIST NTP goes down, the internet doesn’t go down. But atomic clocks drifting does upset many scientific experiments, which would effectively go down for the duration of the outage.

[dead]
Post reply on HN