Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

161–170 of 306 posts

Re: More details about the October 4 outage

#161

Post by cube00 »

joshuamorton wrote: post #143
cube00 wrote: post #128

> We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this. If you correctly design yo…

Can you elaborate on this? There are always going to be security/reliability tradeoffs. Things that fail closed for security reasons will cause slower incident responses. That's unavoidable. Innovation can improve the frontier, but there will always be tradeoffs.

Slower sure, but not five hour slow.

Re: More details about the October 4 outage

#163

Post by tedunangst »

Ansil849 wrote: post #73
educationcto wrote: post #68

Earlier quoted context omitted.

The original error was the network command, but the slower response and lengthy outage was partially due to the physical security they put in place to prevent malicious activity. Any event like this has multiple root causes.

Yes, but the fact that the blogpost concludes on this relatively tangential note (which notably also conveniently allows Facebook to brag about their security measures) and not on the note that their audit code was apparently itself not sufficiently audited, is what makes this deceptive spin.

Seems like appropriate emphasis given how many people yesterday were asking why aren't they back online yet. For every person asking why they deleted their routes there were two people asking why they didn't put them back.

Re: More details about the October 4 outage

#164

Post by bob1029 »

UncleMeat wrote: post #158
agwa wrote: post #150

Earlier quoted context omitted.

> a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust Is that normal at Google? Making people feel bad for an outage doesn't seem consistent with the "blameless postmortem" culture promoted in the SRE book[1]. [1] https://sre.google/sre-book/postmortem-culture/

"Blameless Postmortem" does not mean "No Consequences", even if people often want to interpret it that way. If an organization determines that a disconnect between ground work and a customer's experience is a contributing factor to poor decision making then they might conclude that making engineers more emotionally invested in their customers could be a viable path forward.

Relentless customer service is never going to screw you over in my experience... It pains me that we have to constantly play these games of abstraction between engineer and customer. You are presumably working a job which involves some business and some customer. It is not a fucking daycare. If any of my customers are pissed about their experience, I want to be on the phone with them as soon as humanly possible and I want to hear it myself. Yes, it is a dreadful experience to get bitched at, but it also sharpens your focus like you wouldn't believe when you can't just throw a problem to the guy behind you.

By all means, put the support/enhancement requests through a separate channel+buffer so everyone can actually get work done during the day. But, at no point should an engineer ever be allowed to feel like they don't have to answer to some customer. If you are terrified a junior dev is going to say a naughty phrase to a VIP, then invent an internal customer for them to answer to, and diligently proxy the end customer's sentiment for the engineer's benefit.

Re: More details about the October 4 outage

#165

Post by UncleMeat »

AdamHominem wrote: post #138
vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). Bullshit. I'd believe this if it was not completely impossible for 99.999999% of google "customers" to contact anyone at the company. Or for the decade and a half of p…

Hi there. I'm a Googler and I've directly interfaced with a nontrivial number of customers such that I alone have interfaced with more than 0.000001% of the entire world population.

Re: More details about the October 4 outage

#166

Post by pm2222 »

cranekam wrote: post #116
pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

The wording is a bit unclear (rushed, no doubt) but I expect this means the DNS servers stopped announcing themselves as possible targets for the anycasted IPs Facebook uses for its authoritative DNS [1], since they learned that the network was deemed unhealthy. If they all do that nobody will answer traffic sent to the authoritative DNS IPs and nothing works. [1] See "our authoritative name servers that occupy well…

The anycasted IPs for DNS servers make sense to me and the BGP withdrawal too when the common cases are perhaps one or a few PoPs lost connectivity to the backbone/DC rarely every one of them fails at the time.

I was hoping perhaps the DNS servers at PoPs can be improved by responding with public IPs for other PoPs/DCs and only when that is not available start the BGP withdrawal. Or I can presume the available DNSs at PoPs decrease over time and the remaining ones getting more and more requests until finally every one of them is cut off from the internet.

Re: More details about the October 4 outage

#167

Post by jaywalk »

cnst wrote: post #117
nonbirithm wrote: post #46

DNS seems to be a massive point of failure everywhere, even taking out the tools to help deal with outages themselves. The same thing happened to Azure multiple times in the past, causing complete service outages. Surely there must be some way to better mitigate DNS misconfiguration by now, given the exceptional importance of DNS?

But DNS didn't actually fail. Their design says DNS must go offline if the rest of the network is offline. That's exactly what DNS did. Sounds like their design was wrong, but you can't just blame DNS. DNS worked 100% here as per the task that it was given. > To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication…

I'm not sure the design was even wrong, since the DNS servers being down didn't meaningfully contribute to the outage. The entire Facebook backbone was gone, so even if the DNS servers continued giving out cached responses clients wouldn't be able to connect anyway.

Re: More details about the October 4 outage

#169

Post by colechristensen »

Barrin92 wrote: post #24
HenryKissinger wrote: post #5

> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally Imagine being this person. Tomorrow on /r/tifu.

of course if one person can knock down an entire global system through a trivial mistake the problem is obviously not the person to begin with, but the architecture of the system.

Not really, there’s essentially always a button that blows everything up. Catastrophic failures usually end up being a large set of safety systems malfunctioning which would otherwise prevent the issue when that button is pressed.

But yes, for these types of problem, the ultimate fault is never “that guy Larry is an idiot”, it takes a large team of cooperating mistakes.

Re: More details about the October 4 outage

#170

Post by colechristensen »

antocv wrote: post #66
jimmyvalmer wrote: post #56

tldr; a maintenance query was issued that inexplicably severed FB's data centers from the internet, which unnecessarily caused their DNS servers to mark themselves defunct, which made it all but impossible for their guys to repair the problem from HQ, which compelled them to physically dispatch field units whose progress was stymied by recent increased physical security measures.

> caused their DNS servers to mark themselves defunct This is awkward for me too, why should a DNS server withdraw BGP routes? Design fail.

Designed to handle individual POPs/DCs going down
Post reply on HN