Live data from Hacker News

NTP at NIST Boulder Has Lost Power

lists.nanog.org

151–160 of 218 posts

Re: NTP at NIST Boulder Has Lost Power

#151
post #104
post #80

Earlier quoted context omitted.

Uhh, here's the problem, I'm sort of stuck travelling into the future at a more or less constant rate. I don't know how to stop doing that...

Have you tried asymptotically approaching the speed of light?

I’m quite certain you can approach it in any convenient manner

Re: NTP at NIST Boulder Has Lost Power

#152
post #14

Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.

Notably, we had the marshal fire here 4 years ago and recently Xcel settled for $680M for their role in the fire. So they're probably pretty keen not to be on the hook again

I guess that explains why they had no qualms shutting down half of Boulder's power with a vague time horizon. After losing everything in my fridge, though, they finally turned it back on today.

Re: NTP at NIST Boulder Has Lost Power

#153
post #13
post #5

Can anybody expand on the implications of this? Being unfamiliar with it, it's hard to tell if this is a minor blip that happens all the time, or if it's potentially a major issue that could cause cascading errors equal to the hype of Y2K.

Google has their own fleet of atomic clocks and time servers. So does AWS. So does Microsoft. So does Ubuntu. They're not going to drift enough for months to cause trouble. So the Internet can ride through this, mostly. The main problem will be services that assume at least one of the NIST time servers is up. Somewhere, there's going to be something that won't work right when all the NIST NTP servers are down. But wh…

GPS?

Re: NTP at NIST Boulder Has Lost Power

#154
post #13

Earlier quoted context omitted.

Google has their own fleet of atomic clocks and time servers. So does AWS. So does Microsoft. So does Ubuntu. They're not going to drift enough for months to cause trouble. So the Internet can ride through this, mostly. The main problem will be services that assume at least one of the NIST time servers is up. Somewhere, there's going to be something that won't work right when all the NIST NTP servers are down. But wh…

Ubuntu using atomic clocks would surprise me. Sure they could, but it's not obvious to me why they would spend $$$$ on such. More plausible to me seems that they would be using GPSDO as reference clocks (in this context, about as good as your own atomic clock), iff they were running their own time servers. Google finds only that they are using servers from the NTP Pool Project, which will be using a variety of refere…

atomic clock is not expensive. they have different grades. module level atomic clock cost only $3500.

the NIST hydrogen clock is very expensive and sophisticated.

Re: NTP at NIST Boulder Has Lost Power

#155
post #55

Earlier quoted context omitted.

I never checked it, but how much a typical's pc/server's clock does actually drift over a week or a month? I always thought it's well under a second.

Clocks do drift. Seconds a week is definitely possible. I think there are varying quality of internal clocks in electronic devices, and the cheaper the quality the more drift there is. I think small cheap microcontrollers can drift seconds per day.

cheap microcontrollers use RC oscillators. If they only drift a few seconds a day, that would be an achievement by itself.

RC oscillator is poor enough that early days USB communication would fail if running on RC clock.

Re: NTP at NIST Boulder Has Lost Power

#156
post #14

Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.

Yup, here in Jefferson County - roughly 30 minutes south of Boulder County, we were getting wind gusts around 80mph.

Re: NTP at NIST Boulder Has Lost Power

#157
post #76
post #17

Earlier quoted context omitted.

Atomic clock non-expert here, what does having a fleet of atomic clocks entail and why would the hyperscalers bother?

There's a lot of focus in this thread on the atomic clocks but in most datacenters, they're not actually that important and I'm dubious that the hyperscalers actually maintain a "fleet" of them, in the sense that there are hundreds or thousands of these clocks in their datacenters. The ultimate goal is usually to have a bunch of computers all around the world run synchronised to one clock, within some very small erro…

I don't know about all hyperscalers, but I have knowledge of one of them that has a large enough fleet of atomic frequency standards to warrant dedicated engineering. Several dozen frequency standards at least, possibly low hundreds. Definitely not one per machine, but also not just one per datacenter.

As you say, the goal is to keep the system clocks on the server fleet tightly aligned, to enable things like TrueTime. But also to have sufficient redundancy and long enough holdover in the absence of GNSS (usually due to hardware or firmware failure on the GNSS receivers) that the likelihood of violating the SLA on global time uncertainty is vanishingly small.

The "global" part is what pushes towards having higher end frequency standards, they want to be able to freewheel for O(days) while maintaining low global uncertainty. Drifting a little from external timescales in that scenario is fine, as long as all their machines drift together as an ensemble.

The deployment I know of was originally rubidium frequency standards disciplined by GNSS, but later that got upgraded to cesium standards to increase accuracy and holdover performance. Likely using an "industrial grade" cesium standard that's fairly readily available, very good but not in the same league as the stuff NIST operates.

Re: NTP at NIST Boulder Has Lost Power

#158
post #136

Can anybody speak to the current best practices around running underground power lines? I see these types of articles about above-ground distribution systems from time-to-time, particularly in California. I feel lucky that my area has underground power, but that was installed back in the 1980s. Would it be prohibitively expensive for Boulder’s utility provider to move to underground distribution? I can’t help but thi…

Its very hard to repair and keep track of underground.

https://practical.engineering/blog/2021/9/16/repairing-under...

Re: NTP at NIST Boulder Has Lost Power

#159
We had some fun requesting key for accessing nist time servers. the process is (quoted from website)

NIST currently offers this service free of charge. We require written requests to arrive by U.S. mail or fax containing:

Your organization’s name, physical address, fax number (if desired as a reply method).

One or more point-of-contact personnel or system operators authorized to receive key data and other correspondence: names, phone numbers, email addresses. Up to four static IPv4 network addresses under the user’s control which will be allowed to use the unique key. By special arrangement, additional addresses or address ranges may be requested.

Desired hash function (“key type”). NIST currently supports MD5, SHA1, SHA256, and HMAC-SHA256. Please list any limitations your client software places on key values, if known: maximum length, characters used, or whether hexadecimal key representations are required. If you prefer, please share details about your client software or NTP appliance so we can anticipate key format issues. Desired method for NIST’s reply: U.S. mail, fax, or a secure download service operated by Department of Commerce.

NIST will not use email for sending key data.

ps. there actually seems to be improvement over what they had year ago. they added "secure download service". and previously they had message that nobody assigned to actively monitor mailbox so if you didn't get key, please email us so we will check it

Re: NTP at NIST Boulder Has Lost Power

#160
post #76
post #17

Earlier quoted context omitted.

Atomic clock non-expert here, what does having a fleet of atomic clocks entail and why would the hyperscalers bother?

There's a lot of focus in this thread on the atomic clocks but in most datacenters, they're not actually that important and I'm dubious that the hyperscalers actually maintain a "fleet" of them, in the sense that there are hundreds or thousands of these clocks in their datacenters. The ultimate goal is usually to have a bunch of computers all around the world run synchronised to one clock, within some very small erro…

> There's a lot of focus in this thread on the atomic clocks but in most datacenters, they're not actually that important and I'm dubious that the hyperscalers actually maintain a "fleet" of them, in the sense that there are hundreds or thousands of these clocks in their datacenters.

I mean, fleets come in all sizes; but if you put one atomic reference in each AZ of each datacenter, there's a fleet. Maybe the references aren't great at distributing time, so you add a few NTP distributors per datacenter too and your fleet is a little bigger. Google's got 42 regions in GCP, so they've got a case for hundreds of machines for time (plus they've invested in spanner which has some pretty strict needs); other clouds are likely similar.

Post reply on HN