Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NTP at NIST Boulder Has Lost Power

lists.nanog.org

Post reply on HN
Print view View original

111–120 of 218 posts

Re: NTP at NIST Boulder Has Lost Power

#111

Post by grepfru_it »

sgnelson wrote: post #90

One question I have is did DOGE decisions have anything to do with this? Because I know they took knives to NIST.

Residents and some businesses of Boulder have been without power since Tuesday. There was an issue about 10 years ago which caused 1000 homes to burn down and the power company was found liable. They change their actions. Then during the next high wind event, the power company preemptively cut power and businesses sued them for loss of revenue. Now the power company is playing it safe and turning off power to residents and keeping downtown businesses powered.

Maybe their generator failing was DOGE related, but wouldn’t have happened if state level shenanigans were better handled

Re: NTP at NIST Boulder Has Lost Power

#112

Post by layer8 »

Roark66 wrote: post #55
ThrowawayTestr wrote: post #11

Earlier quoted context omitted.

If your computer was using it as your time server and you didn't have alternatives configured your clock my have drifted a few seconds.

I never checked it, but how much a typical's pc/server's clock does actually drift over a week or a month? I always thought it's well under a second.

Several seconds per week is normal. Oscillator accuracy is roughly on the order of 10 PPM, which would correspond to 6 seconds per week.

Re: NTP at NIST Boulder Has Lost Power

#113

Post by tpoindex »

IncreasePosts wrote: post #78
arn3n wrote: post #14

Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.

Notably, we had the marshal fire here 4 years ago and recently Xcel settled for $680M for their role in the fire. So they're probably pretty keen not to be on the hook again

For more background on the Marshal Fire of Dec. 2021: https://en.wikipedia.org/wiki/Marshall_Fire

tl;dr - the fire destroyed over 1,000 homes, two deaths. The local electrical utility, Xcel, was found as a contributing cause from sparking power lines during a strong wind storm. As a result, electrical utilities now cut power to affected areas during strong winds.

Re: NTP at NIST Boulder Has Lost Power

#114

Post by the_gipsy »

cramcgrab wrote: post #95
sgnelson wrote: post #90

One question I have is did DOGE decisions have anything to do with this? Because I know they took knives to NIST.

Actually DOGE involvement at the highest level would have resulted in Tesla solar and Tesla powerwall battery backups.

Any day now

Re: NTP at NIST Boulder Has Lost Power

#115

Post by throw0101c »

geerlingguy wrote: post #108
idiotsecant wrote: post #91

Earlier quoted context omitted.

If your application is so critical that NTP timing loss causes disaster and your holdover fails in less than a day and you aren't generating your own via gps, you are incompetent, full stop

And if things are that critical, you might have other references besides just GPS...

E.g.,

* https://www.meinbergglobal.com/english/products/

Re: NTP at NIST Boulder Has Lost Power

#117

Post by throw0101c »

PunchyHamster wrote: post #64
Vosporos wrote: post #56

Earlier quoted context omitted.

One (amongst many) NTP server going down creates less issues than an NTP server spreading wrong time.

technically if you have 3 or more sources that would be caught; NTP protocol was designed for that eventuality

> technically if you have 3 or more sources that would be caught; NTP protocol was designed for that eventuality

Either go with one clock in your NTPd/Chrony configuration, or ≥4.

Yes, if you have 3 they can triangulate, but if one goes offline now you have 2 with no tie-breaker. If you have (at least) 4 servers, then one can go away and triangulation / sanity-checking can still occur with the 3 remaining.

Re: NTP at NIST Boulder Has Lost Power

#118

Post by throw0101c »

cramcgrab wrote: post #95
sgnelson wrote: post #90

One question I have is did DOGE decisions have anything to do with this? Because I know they took knives to NIST.

Actually DOGE involvement at the highest level would have resulted in Tesla solar and Tesla powerwall battery backups.

> […] Tesla solar and Tesla powerwall battery backups.

Don't forget Solar Roof.

Re: NTP at NIST Boulder Has Lost Power

#119

Post by misnome »

adastra22 wrote: post #69
misnome wrote: post #43

Earlier quoted context omitted.

All these use at least GPS for timing

No, they don’t. GPS is orders of magnitude less reliable than the most up to date metric time synchronization over fixed topology fiber links.

I wonder why we bothered building GPS signal waveguides into the bottom of a mine then. Clearly we should have consulted the experts of hacker news first.

Losing NTP for a day is going to affect fuck-all.

Re: NTP at NIST Boulder Has Lost Power

#120

Post by joncrane »

jhart99 wrote: post #52
cdfuller wrote: post #5

Can anybody expand on the implications of this? Being unfamiliar with it, it's hard to tell if this is a minor blip that happens all the time, or if it's potentially a major issue that could cause cascading errors equal to the hype of Y2K.

NIST maintains several time standards. Gaithersburg MD is still up and I assume Hawaii is as well. Other than potential damage to equipment from loss of power (turbo molecular vacuum pumps and oil diffusion pumps might end up failing in interesting ways if not shut down properly) it will just take some time for the clocks to be recalibrated against the other NIST standards.

Gaithersburg went down on December 8th, is there confirmation that it's fully functional again?
Post reply on HN