[deleted]
More details about the October 4 outage
101–110 of 306 posts
Re: More details about the October 4 outage
#102Will somebody lose a job over this?
They shouldn't. If FB has a proper PMA culture in place, you figure out why the processes in place didn't work that allowed this kind of change to happen, so more testing, etc. Should be a blameless exercise.
Re: More details about the October 4 outage
#103Earlier quoted context omitted.
The original error was the network command, but the slower response and lengthy outage was partially due to the physical security they put in place to prevent malicious activity. Any event like this has multiple root causes.
Yes, but the fact that the blogpost concludes on this relatively tangential note (which notably also conveniently allows Facebook to brag about their security measures) and not on the note that their audit code was apparently itself not sufficiently audited, is what makes this deceptive spin.
I agree that there's an awkward emphasis on how FB prioritizes security and privacy but nothing is deceptive here. Had the audit bug not subsequently cut off access to internal tools and remote regions it would be easy to revert. Had there not been a global outage nobody would have known that the process for getting access in an emergency was too slow.
Huge events like this always have many factors that have to line up just right. To insist that the one and only true cause was a bug in the auditing system is reductive.
Re: More details about the October 4 outage
#104> To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.
No, it's (clearly) not a guaranteed indication of that. Logic fail. Infrastructure tools at that scale need to handle all possible causes of test failures. "Is the internet down or only the few sites I'm testing?" is a classic network monitoring script issue.
Re: More details about the October 4 outage
#105> We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this. From here on out, our job…
Simple Testing Can Prevent Most Critical Failures[1], "We found the majority of catastrophic failures could easily have been prevented by performing simple testing on error handling code – the last line of defense – even without an understanding of the software design." 1 https://www.eecg.utoronto.ca/~yuan/papers/failure_analysis_o...
Having a separate testing instance of the internet might not be practical. How exactly would you test such a change? Simulating the effect of router commands is a very daunting challenge.
Re: More details about the October 4 outage
#106Was this a way to delete a lot of evidence before shit really hit the fan?' After reading this, I can't help but feel this was a calculated move. It gives FB a chance to hijack media attention from the whistleblower. It gives them a chance to show the average peson, 'hey, we make mistakes and we have a review process to improve our systems'. The timing is too perfect if you ask me.
I'm not usually that cynical, but the timing of it combined with facebook's lengthy abusive relationship with customers' privacy (and what kind of company morals that implies) makes me think that it's definitely a possibility.
The testimony in front of congress and the reaction is just making me feel even more like this was a calculated move or internal sabotage.
Re: More details about the October 4 outage
#107Earlier quoted context omitted.
How do you avoid circular dependencies on an out-of-band-network? Seems like the choice is between a circular dependency, or turtles all the way down.
How do you go from "have a separate access method that doesn't depend on your main system" to "turtles all the way down"? The secondary access is allowed to have dependencies, just not on your network.
And if the secondary access fails, then what? Backup systems are not reliable 100% of the time.
Re: More details about the October 4 outage
#108So someone ran "clear mpls lsp" instead of "show mpls lsp"?
Seems like it. It's kinda like typing hostname and accidentally poking your yubikey (Not that I've done that...) or the date command that both let's you set the date and format the date
Re: More details about the October 4 outage
#109I want to know what happened to the poor engineer who issued the command?
He will be promoted to street dweller while his managers will fail up.
Actually he was promoted to C-Level, CII, Chief Imperial Intern ‘For Life’.
It’s a great accomplishment to be be fair, comes with a lifetime weekly stipend and access to whatever Frontend books/courses you need to be a great web developer.
Will never touch ops again.
Re: More details about the October 4 outage
#110Note that contrary to popular reports, DNS was NOT to blame for this outage — for once DNS worked exactly as per per the spec, design and configuration:
> To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.