Interesting that they think fluctuations of tens of megawatts would risk electrical systems. If the equipment was handling that much continuous load, wouldn't it also easily handle the resumption of the same load? Also I totally did not understand how power usage would affect caches.
More details about the October 4 outage
251–260 of 306 posts
Re: More details about the October 4 outage
#252Earlier quoted context omitted.
The buggy audit tool was probably made by a human too, though.
But reviewed by other humans. At some count, a collective human error becomes a system error.
- Me, 2021
Re: More details about the October 4 outage
#253Incidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.
Re: More details about the October 4 outage
#254Re: More details about the October 4 outage
#255> Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk. Interesting that they think fluctuations of tens of megawatts would risk electrical systems. If the equipment was handling that much continuous load, wouldn't it also easily handle the resumption of the sa…
Re: More details about the October 4 outage
#256> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.
> I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error. Wouldn't human error reflect extremely poorly on the company though? I mean, for human error to be the root cause of this mega-outage, that would imply that the company's infrastructure and operational and security practices were so ineffective that a single person scre…
For a modern example, look for information on Symantec's "A tough day as leaders" in which they try to blame an incident that's clearly a result of at least incompetence by senior management on a single person who they've just fired. This is part of the sequence of events that leads to Symantec no longer being a trusted root CA. You won't find that actual post by Symantec because (of course) once they realised it wasn't doing what they wanted they deleted it, but you can find copies and references to it.
For much older examples, look at the early history of the railway in most of the world. Train crashes, blame the (often dead in the crash and thus unable to defend themselves) train driver, hint that they may have been drunk and were certainly incompetent. Owners carry on profiting from unsafe railway and needn't spend any money making it safer.
Re: More details about the October 4 outage
#257Why were all BGP routes advertised from same set of servers in same DC which were pointed by ALL fb owned domains?
Some of the speculation I have seen was that this was an attempt to cut connections to one internet exchange (IXP) point, possibly resulting in cutting connection to all IXPs. But ,as I said, this is speculation. It'd take a more thorough understanding of how the automation looked prior to the incident, the change hat was trying to be made, and eth like, to say something sensible.
Re: More details about the October 4 outage
#258Earlier quoted context omitted.
Can you elaborate on this? There are always going to be security/reliability tradeoffs. Things that fail closed for security reasons will cause slower incident responses. That's unavoidable. Innovation can improve the frontier, but there will always be tradeoffs.
Slower sure, but not five hour slow.
You don't want the data centre staff to be able to change configurations (security), so once something requires hands-on changing, you are definitely into the "move people around" stage of recovery and it WILL be slow.
Re: More details about the October 4 outage
#259Earlier quoted context omitted.
Ah yea I meant brownouts for other people haha. I figure Facebook can handle their own electrical stability just fine
Is there any liability if Facebook had brought everything up at once and caused brownouts? Seems like it would be some form of negligence on their part harming a shared resource, but I don't know if there's any laws or contract terms with the power company that require them to pay if they mess up like that.
There are a lot of automated fail-safes on this, and apparently larger industry (which a datacenter is as well) will get disconnected from the grid automatically in emergency situations before they drop residential areas. But in the end they will drop one by one everything they need to keep the larger grid running. It's not even a networked "smart" management system, the distribution points automatically react to voltage and frequency drops and they're set up to break some things like industry earlier than others.
Re: More details about the October 4 outage
#260So someone ran "clear mpls lsp" instead of "show mpls lsp"?
My guess : he executed the command from shell history. Commands look similar enough and he hit Enter too quickly
Would fit muscle memory, but if that wasn't caught by the automated tool they have some work to do.