Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

131–140 of 306 posts

Re: More details about the October 4 outage

#131

Post by fanf2 »

pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

Facebook’s authoritative DNS servers are at the borders between Facebook’s backbone and the rest of the Internet.

Re: More details about the October 4 outage

#132

Post by mumblemumble »

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.

I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

Re: More details about the October 4 outage

#133

Post by Ansil849 »

cranekam wrote: post #103
Ansil849 wrote: post #73

Earlier quoted context omitted.

Yes, but the fact that the blogpost concludes on this relatively tangential note (which notably also conveniently allows Facebook to brag about their security measures) and not on the note that their audit code was apparently itself not sufficiently audited, is what makes this deceptive spin.

I agree that there's an awkward emphasis on how FB prioritizes security and privacy but nothing is deceptive here. Had the audit bug not subsequently cut off access to internal tools and remote regions it would be easy to revert. Had there not been a global outage nobody would have known that the process for getting access in an emergency was too slow. Huge events like this always have many factors that have to line…

> I agree that there's an awkward emphasis on how FB prioritizes security and privacy but nothing is deceptive here.

I guess deceptive was the wrong word, so whatever's the term for "awkward emphasis" :).

Re: More details about the October 4 outage

#134

Post by fabian2k »

narrator wrote: post #90
tantalor wrote: post #49

Earlier quoted context omitted.

But the FB outage was not a configuration change. > a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network

This is kind of like Chernobyl where they were testing to see how hot they could run the reactor to see how much power it could generate. Then things went sideways.

As already said the test was about something entirely different. And the dangerous part was not the test itself, but the way they delayed the test and then continued to perform it despite the reactor being in a problematic state and the night shift being on duty, who were not trained on this test. The main problem was that they ran the reactor at reduced power long enough to have significant xenon poisoning, and then put the reactor at the brink when they tried to actually run the test under these unsafe conditions.

Re: More details about the October 4 outage

#135

Post by bastardoperator »

flutas wrote: post #112
Barrin92 wrote: post #24

Earlier quoted context omitted.

of course if one person can knock down an entire global system through a trivial mistake the problem is obviously not the person to begin with, but the architecture of the system.

Or the fact that there was a bug in the tool that should have prevented this. > Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.

So this is really two peoples fault. One for issuing the command the other for introducing the bug in the audit tool.

Re: More details about the October 4 outage

#136

Post by xphos »

I don't know the -f in rm -rf isn't a bug xD. I feel sorry for the poor engineer who fat fingered the command. It definitely highlights an anti-pattern in the command line but the fact that, that singular console had the power to effect the entire network highlights an "interesting" design choice indeed.

Re: More details about the October 4 outage

#137

Post by tobr »

mumblemumble wrote: post #132

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

The buggy audit tool was probably made by a human too, though.

Re: More details about the October 4 outage

#138

Post by AdamHominem »

vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust).

Bullshit.

I'd believe this if it was not completely impossible for 99.999999% of google "customers" to contact anyone at the company. Or for the decade and a half of personal and professional observations of people getting fucked over by google and having absolutely nobody they could contact to try and resolve the situation.

You googlers can't even disdain yourselves to talk to other workers at the company who are in a caste lower than you.

The fundamental problem googlers have is that they all think they're so smart/good at what they do, it just doesn't seem to occur that they could have possibly screwed something up, or something could go wrong or break, or someone might need help in a way your help page authors didn't anticipate...and people might need to get ahold of an actual human to say "shit's broke, yo." Or worse, none of you give a shit. The company certainly doesn't. When you've got near monopoly and have your fingers in every single aspect of the internet, you don't need to care about fucking your customers over.

I cannot legitimately name a single google product that I, or anyone I know, likes or wants to use. We just don't have a choice because of your market dominance.

Re: More details about the October 4 outage

#139

Post by silicon2401 »

vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> leadership read out emails from customers telling us how we let them down and lost their trust).

That's amazing. I would never have expected my feedback to a company to actually be read, let alone taken seriously. Hopefully more companies do this than I thought.

Re: More details about the October 4 outage

#140

Post by Ajedi32 »

oarabbus_ wrote: post #107
detaro wrote: post #89

Earlier quoted context omitted.

How do you go from "have a separate access method that doesn't depend on your main system" to "turtles all the way down"? The secondary access is allowed to have dependencies, just not on your network.

And if the secondary access fails, then what? Backup systems are not reliable 100% of the time.

Then you're SOL. What's your point? The backup might fail, so don't have a backup? I don't understand what you're trying to say.
Post reply on HN