Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

251–260 of 306 posts

Re: More details about the October 4 outage

#251
> Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk.

Interesting that they think fluctuations of tens of megawatts would risk electrical systems. If the equipment was handling that much continuous load, wouldn't it also easily handle the resumption of the same load? Also I totally did not understand how power usage would affect caches.

Re: More details about the October 4 outage

#252
post #186
post #137

Earlier quoted context omitted.

The buggy audit tool was probably made by a human too, though.

But reviewed by other humans. At some count, a collective human error becomes a system error.

System complexity is just a way to avoid blaming individual humans when an error occurs.

- Me, 2021

Re: More details about the October 4 outage

#253

Incidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.

On Instagram I was even able to "like" posts while the outage was in effect. Not sure if the app replayed those when the service came back.

Re: More details about the October 4 outage

#254
When facebook is directly peering to so many other ASes, why would they not have static routes in place for those direct links? Why run BGP for that? It's not like there is going to be a better route than the direct link. If the link goes down, then you can rely on BGP to reroute.

Re: More details about the October 4 outage

#255

> Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk. Interesting that they think fluctuations of tens of megawatts would risk electrical systems. If the equipment was handling that much continuous load, wouldn't it also easily handle the resumption of the sa…

Continuous load is very different from a sudden 0 to 100 increase.

Re: More details about the October 4 outage

#256
post #142

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

> I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error. Wouldn't human error reflect extremely poorly on the company though? I mean, for human error to be the root cause of this mega-outage, that would imply that the company's infrastructure and operational and security practices were so ineffective that a single person scre…

Organisations have a bad habit of using "human error" to blame systemic problems whose true root cause is inadequate leadership on individual low level employees. So, we're glad to see Facebook didn't try this shitty practice.

For a modern example, look for information on Symantec's "A tough day as leaders" in which they try to blame an incident that's clearly a result of at least incompetence by senior management on a single person who they've just fired. This is part of the sequence of events that leads to Symantec no longer being a trusted root CA. You won't find that actual post by Symantec because (of course) once they realised it wasn't doing what they wanted they deleted it, but you can find copies and references to it.

For much older examples, look at the early history of the railway in most of the world. Train crashes, blame the (often dead in the crash and thus unable to defend themselves) train driver, hint that they may have been drunk and were certainly incompetent. Owners carry on profiting from unsafe railway and needn't spend any money making it safer.

Re: More details about the October 4 outage

#257

Why were all BGP routes advertised from same set of servers in same DC which were pointed by ALL fb owned domains?

I don't think they were. But, without knowing more specifically what happened, it is hard to speculate.

Some of the speculation I have seen was that this was an attempt to cut connections to one internet exchange (IXP) point, possibly resulting in cutting connection to all IXPs. But ,as I said, this is speculation. It'd take a more thorough understanding of how the automation looked prior to the incident, the change hat was trying to be made, and eth like, to say something sensible.

Re: More details about the October 4 outage

#258
post #161

Earlier quoted context omitted.

Can you elaborate on this? There are always going to be security/reliability tradeoffs. Things that fail closed for security reasons will cause slower incident responses. That's unavoidable. Innovation can improve the frontier, but there will always be tradeoffs.

Slower sure, but not five hour slow.

The moment you need to start moving people around, you are into "hours" territory of recovery.

You don't want the data centre staff to be able to change configurations (security), so once something requires hands-on changing, you are definitely into the "move people around" stage of recovery and it WILL be slow.

Re: More details about the October 4 outage

#259
post #227
post #222

Earlier quoted context omitted.

Ah yea I meant brownouts for other people haha. I figure Facebook can handle their own electrical stability just fine

Is there any liability if Facebook had brought everything up at once and caused brownouts? Seems like it would be some form of negligence on their part harming a shared resource, but I don't know if there's any laws or contract terms with the power company that require them to pay if they mess up like that.

My girlfriend works in a large grid operator (in Europe). According to her there are lots of regulations and contracts on the grid operators about how they must handle reliability. So it's unlikely that Facebook would be liable if this took down half the country, because then it was the grid operator not living up to their agreements on reliability.

There are a lot of automated fail-safes on this, and apparently larger industry (which a datacenter is as well) will get disconnected from the grid automatically in emergency situations before they drop residential areas. But in the end they will drop one by one everything they need to keep the larger grid running. It's not even a networked "smart" management system, the distribution points automatically react to voltage and frequency drops and they're set up to break some things like industry earlier than others.

Re: More details about the October 4 outage

#260
post #162
post #15

So someone ran "clear mpls lsp" instead of "show mpls lsp"?

My guess : he executed the command from shell history. Commands look similar enough and he hit Enter too quickly

cltr+r mpls [enter]

Would fit muscle memory, but if that wasn't caught by the automated tool they have some work to do.

Post reply on HN