Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

221–230 of 306 posts

Re: More details about the October 4 outage

#221

Post by mjevans »

Johnny555 wrote: post #201
cesarb wrote: post #179

Earlier quoted context omitted.

The key word is "suddenly". In the electricity grid, demand and generation must always be precisely matched (otherwise, things burn up). This is done by generators automatically ramping up or down whenever the load changes. But most generators cannot change their output instantly; depending on the type of generator, it can take several minutes or even hours to respond to a large change in the demand. Now consider tha…

I get that lots of servers can add up to lots of power, but what is a "lot"? Is 1MW really enough demand to destabilize a regional power grid?

If it's all at once at the end of one leg and unplanned? Yes.

The question is somewhat similar to a thought experiment. If a ship is docked and loading cargo, is it a good idea to use all the cranes to suddenly fill up one outer side of the ship?

Re: More details about the October 4 outage

#222

Post by dmoy »

Johnny555 wrote: post #43
plorg wrote: post #17

Earlier quoted context omitted.

Brownouts is probably the most proximate concern - a sudden increase in demand will draw down the system frequency in the vicinity, and if there aren't generation units close enough or with enough dispatchable capacity there's a small chance they would trip a protective breaker. A person I know on the power grid side said at one data center there were step functions when FB went down and then when it came up, equal t…

But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly. Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?

Ah yea I meant brownouts for other people haha. I figure Facebook can handle their own electrical stability just fine

Re: More details about the October 4 outage

#223

Post by idiotsecant »

cesarb wrote: post #179
Johnny555 wrote: post #50

Earlier quoted context omitted.

I don't see how suddenly running more traffic is going to trip datacenter breakers -- I could see how flipping on power to an entire datacenter's worth of servers could cause a spike in electrical demand that the power infrastructure can't handle, but if suddently running CPU's at 100% trips breakers, then it seems like that power infrastructure is undersized? This isn't a case where servers were powered off, they we…

The key word is "suddenly". In the electricity grid, demand and generation must always be precisely matched (otherwise, things burn up). This is done by generators automatically ramping up or down whenever the load changes. But most generators cannot change their output instantly; depending on the type of generator, it can take several minutes or even hours to respond to a large change in the demand. Now consider tha…

>generated; while the generators are catching up to that, the voltage drops, which means the current increases to compensate...

Close- you won't see an increase in load of a synchronous machine operating at constant throttle manifest as a voltage sag, you'll see it manifest as a decrease in frequency (this generators literally slow down, like a guy on a bike going uphill). Voltage sags are more related to transmission line phenomenon.

Re: More details about the October 4 outage

#225

Post by idiotsecant »

Johnny555 wrote: post #201
cesarb wrote: post #179

Earlier quoted context omitted.

The key word is "suddenly". In the electricity grid, demand and generation must always be precisely matched (otherwise, things burn up). This is done by generators automatically ramping up or down whenever the load changes. But most generators cannot change their output instantly; depending on the type of generator, it can take several minutes or even hours to respond to a large change in the demand. Now consider tha…

I get that lots of servers can add up to lots of power, but what is a "lot"? Is 1MW really enough demand to destabilize a regional power grid?

No. All balancing authorities are required to keep a certain amount of "spinning reserve" available for fast adjustments like this. But if I do it and the next guy does it and a transmission like is down and...etc

A lot of horror stories start that way.

Re: More details about the October 4 outage

#226

Post by karmakaze »

cnst wrote: post #110

Note that contrary to popular reports, DNS was NOT to blame for this outage — for once DNS worked exactly as per per the spec, design and configuration: > To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.

Not the first cause, but involved. Before reading I expected to see some combination of (1) automation (2) DNS (3) BGP. I didn't expect to see all three and the special automatically disconnect the backbone from the internet with no other way for senior tech staff to get to the backbone, not even a secure dial-up console.

I think the general lesson here is for each thing you automate, assume that it can act in error and have another manual way to do what the automatic action prevents.

Re: More details about the October 4 outage

#227

Post by ziml77 »

dmoy wrote: post #222
Johnny555 wrote: post #43

Earlier quoted context omitted.

But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly. Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?

Ah yea I meant brownouts for other people haha. I figure Facebook can handle their own electrical stability just fine

Is there any liability if Facebook had brought everything up at once and caused brownouts? Seems like it would be some form of negligence on their part harming a shared resource, but I don't know if there's any laws or contract terms with the power company that require them to pay if they mess up like that.

Re: More details about the October 4 outage

#228

Post by UncleMeat »

agwa wrote: post #200
UncleMeat wrote: post #158

Earlier quoted context omitted.

"Blameless Postmortem" does not mean "No Consequences", even if people often want to interpret it that way. If an organization determines that a disconnect between ground work and a customer's experience is a contributing factor to poor decision making then they might conclude that making engineers more emotionally invested in their customers could be a viable path forward.

From the SRE book: "For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior. A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had. If a culture of finger pointing and shaming individuals or te…

I work at Google and have written more than a few blameless postmortems. You don't need to quote things to me.

Is what was described above "finger pointing or shaming"? I don't work in TI so I didn't experience this meeting but it doesn't seem like it is. It also doesn't sound to me like this was the only outcome, where the execs just wagged their fingers at engineers and called it a day. Of course there'd be all sorts of process improvements derived from an understanding of the various system causes that led to an outage.

Re: More details about the October 4 outage

#229

Post by BurningFrog »

tobr wrote: post #137
mumblemumble wrote: post #132

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

The buggy audit tool was probably made by a human too, though.

It's humans all the way down!

Re: More details about the October 4 outage

#230

Post by 1vuio0pswjnm7 »

"Those translation queries are answered by our authoritative name servers that occupy well known IP addresses themselves, which in turn are advertised to the rest of the internet via another protocol called the border gateway protocol (BGP)."

"To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection."

Correct me if I am wrong, but here "DNS servers" means the computers, not the software running on them, i.e., each computer is running both DNS software and a BGP daemon. I am not aware of DNS server software that disables BGP advertisements but a BGP daemon could do it.

For example, a BGP daemon like ExaBGP can execute a DNS query, check the output and disable advertisements if the query fails.

https://github.com/Exa-Networks/exabgp

Post reply on HN