Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NTP at NIST Boulder Has Lost Power

lists.nanog.org

Post reply on HN
Print view View original

41–50 of 218 posts

Re: NTP at NIST Boulder Has Lost Power

#41

Post by Maxious »

lovich wrote: post #12

This was an NTP 0 server right? What is the actual failback mechanism when that level of NTP server fails? This is some level of eldritch magic that I am aware of, but not familiar with but am interested in learning.

There's two other sites for the time.nist.gov service so it'll be okay.

Probably more interesting is how you get a tier 0 site back in sync - NIST rents out these cyberpunk looking units you can use to get your local frequency standards up to scratch for ~$700/month https://www.nist.gov/programs-projects/frequency-measurement...

Re: NTP at NIST Boulder Has Lost Power

#42

Post by jeffrallen »

autarch wrote: post #10
cdfuller wrote: post #5

Can anybody expand on the implications of this? Being unfamiliar with it, it's hard to tell if this is a minor blip that happens all the time, or if it's potentially a major issue that could cause cascading errors equal to the hype of Y2K.

Time travel is extremely dangerous right now. I highly recommend deferring time travel plans except for extreme temporal emergencies.

Would traveling to the past in order to put in place a preemptive fix for this outage be wise or dangerous?

Asking for a friend.

Re: NTP at NIST Boulder Has Lost Power

#43

Post by misnome »

adastra22 wrote: post #21
_zoltan_ wrote: post #19

Earlier quoted context omitted.

could you list 3 things that you think are more important than the internet? (I know the internet is going to be fine; I just want to understand what you think ranks higher globally...)

Mostly scientific stuff like astronomical observations — e.g. did this event observed at one telescope coincide with neutrinos detected at this other observatory. Note I didn’t say they are more important than the Internet. That’s a value judgement in any case. I said that NIST level 0 NTp servers are more important to these use cases than they are to the Internet.

All these use at least GPS for timing

Re: NTP at NIST Boulder Has Lost Power

#44

Post by riedel »

Gabrys1 wrote: post #20
genidoi wrote: post #17

Earlier quoted context omitted.

Atomic clock non-expert here, what does having a fleet of atomic clocks entail and why would the hyperscalers bother?

Having clocks synchronized between your servers is extremely useful. For example, having a guarantee that the timestamp of arrival of a packet (measured by the clock on the destination) is ALWAYS bigger than the timestamp recorded by the sender is a huge win, especially for things like database scaling. For this though you need to go beyond NTP into PTP which is still usually based on GPS time and atomic clocks

Actually interesting to think about what UTC actually means and there is seems to be no absolute source of truth [0]. I guess the worry is not that much about the NTP servers (for which people anyways should configure fail overs) but the clocks themselves.

[0] https://www.septentrio.com/en/learn-more/insights/how-gps-br...

Re: NTP at NIST Boulder Has Lost Power

#45

Post by amelius »

TylerE wrote: post #37
Maxion wrote: post #33

Earlier quoted context omitted.

Somewhat interesting that they themselves don't have access to the site. You'd think there would have been some disaster plans put in place?

Step One of most disaster plans is not to create a second emergency.

But can't NTP server downtime cause a disaster?

Re: NTP at NIST Boulder Has Lost Power

#46

Post by DamonHD »

So far I think I'm still seeing one of them in my peers list for my public-ish NTP server:

         remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
    +time-e-b.nist.g .NIST.           1 u  372 1024  377  125.260    1.314   0.280

Re: NTP at NIST Boulder Has Lost Power

#47

Post by amelius »

This makes me wonder, if you take the average time of all wristwatches on the planet, accounting for timezones and throwing out outliers, how close would you get to NTP time?

And how many randomly chosen wristwatches would you need to get anything reasonable?

Re: NTP at NIST Boulder Has Lost Power

#48

Post by lovich »

Maxious wrote: post #41
lovich wrote: post #12

This was an NTP 0 server right? What is the actual failback mechanism when that level of NTP server fails? This is some level of eldritch magic that I am aware of, but not familiar with but am interested in learning.

There's two other sites for the time.nist.gov service so it'll be okay. Probably more interesting is how you get a tier 0 site back in sync - NIST rents out these cyberpunk looking units you can use to get your local frequency standards up to scratch for ~$700/month https://www.nist.gov/programs-projects/frequency-measurement...

What happens in the event all the sites for time.nist.gov go down? is it included in the spec?

Also thank you for that link, this is exactly the kind of esoteric knowledge that I enjoy learning about

Re: NTP at NIST Boulder Has Lost Power

#49

Post by ssl-3 »

Maxion wrote: post #33
arn3n wrote: post #14

Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.

Somewhat interesting that they themselves don't have access to the site. You'd think there would have been some disaster plans put in place?

Maybe this is the disaster plan: There's not a smouldering hole where NIST's Boulder facility used to be, and it will be operational again soon enough.

There's no present need for important hard-to-replace sciencey-dudes to go into the shop (which is probably both cold, and dark, and may have other problems that make it unsafe: it's deliberately closed) to futz around with the the time machines.

We still have other NTP clocks. Spooky-accurate clocks that the public can get to, even, like just up the road at NIST in Fort Collins (where WWVB lives, and which is currently up), and in Maryland.

This is just one set.

And beyond that, we've also got clocks in GPS satellites orbiting, and a whole world of low-stratum NTP servers that distribute that time on the network. (I have one such GPS-backed NTP server on the shelf behind me; there's not much to it.)

And the orbital GPS clocks are controlled by the US Navy, not NIST.

So there's redundancy in distribution, and also control, and some of the clocks aren't even on the Earth.

Some people may be bit by this if their systems rely on only one NTP server, or only on the subset of them that are down.

And if we're following section 3.2 of RFC 8633 and using multiple diverse NTP sources for our important stuff, then this event (while certainly interesting!) is not presently an issue at all.

Re: NTP at NIST Boulder Has Lost Power

#50

Post by DamonHD »

DamonHD wrote: post #46

So far I think I'm still seeing one of them in my peers list for my public-ish NTP server: remote refid st t when poll reach delay offset jitter ============================================================================== +time-e-b.nist.g .NIST. 1 u 372 1024 377 125.260 1.314 0.280

...and maybe it's gone:

    #time-e-b.nist.g .NIST.           1 u 1071 1024  377  125.260    1.314   0.280
Post reply on HN