More details about the October 4 outage • Hacker News Board

Re: More details about the October 4 outage

#181

Post by MrDunham » Tue, Oct 05, 2021, 8:25 PM UTC

dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

I’m very close with someone who works at a FB data center and was discussing this exact issue.

I can only speak to one problem I know of (and am rather sure I can share): a spike might trip a bunch of breakers at the data center.

BUT, unlike me at home, FBs policy is to never flip a circuit back on until you’re positive of the root cause of said trip.

By itself that could compound issues and delay ramp up time as they’d work to be sure no electrical components actually sorted/blew/etc. A potentially time sucking task given these buildings could be measured in whole units of football fields.

Re: More details about the October 4 outage

#182

Post by cnst » Tue, Oct 05, 2021, 8:29 PM UTC

jaywalk wrote: post #167

cnst wrote: post #117

Earlier quoted context omitted.

But DNS didn't actually fail. Their design says DNS must go offline if the rest of the network is offline. That's exactly what DNS did. Sounds like their design was wrong, but you can't just blame DNS. DNS worked 100% here as per the task that it was given. > To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication…

I'm not sure the design was even wrong, since the DNS servers being down didn't meaningfully contribute to the outage. The entire Facebook backbone was gone, so even if the DNS servers continued giving out cached responses clients wouldn't be able to connect anyway.

Exactly. And it would actually be worse, because the clients would have to wait for a timeout, instead of simply returning a name error right away.

Re: More details about the October 4 outage

#183

Post by quietbritishjim » Tue, Oct 05, 2021, 8:30 PM UTC

vitus wrote: post #58

tantalor wrote: post #49

Earlier quoted context omitted.

But the FB outage was not a configuration change. > a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network

From yesterday's post: "Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. ... Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear that there was no malicious activity behind this outage — its root cause…

I'm inclined to believe the later post as they've had more time to assess the details. I think the point of the earlier post is really to say "we weren't hacked!" but they didn't want to use exactly that language.

Re: More details about the October 4 outage

#184

Post by grumple » Tue, Oct 05, 2021, 8:30 PM UTC

vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

The most remarkable thing about this is learning that anyone at Google read an email from a customer. Given the automated responses to complaints of account shutdowns, or complaints about app store rejections, etc, this is pretty surprising.

Re: More details about the October 4 outage

#185

Post by bilater » Tue, Oct 05, 2021, 8:31 PM UTC

pjscott wrote: post #168

bilater wrote: post #53

I want to know what happened to the poor engineer who issued the command?

The blog post is putting the blame on a bug in the tooling which should have made the command impossible to issue, which is exactly where the blame ought to go.

Still I'd hate to be the first 'Why' of a multi-billion dollar outage :D

Re: More details about the October 4 outage

#186

Post by jhrmnn » Tue, Oct 05, 2021, 8:32 PM UTC

tobr wrote: post #137

mumblemumble wrote: post #132

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

The buggy audit tool was probably made by a human too, though.

But reviewed by other humans. At some count, a collective human error becomes a system error.

Re: More details about the October 4 outage

#187

Post by cnst » Tue, Oct 05, 2021, 8:34 PM UTC

Xylakant wrote: post #175

jaywalk wrote: post #167

Earlier quoted context omitted.

I'm not sure the design was even wrong, since the DNS servers being down didn't meaningfully contribute to the outage. The entire Facebook backbone was gone, so even if the DNS servers continued giving out cached responses clients wouldn't be able to connect anyway.

DNS being down instead of returning an unreachable destination did increase load for other DNS resolvers though since empty results cannot be cached and clients continued to retry. This made the outage affect others.

Source?

DNS errors are actually still cached; it's something that has been debunked by DJB like a couple of decades ago, give or take:

http://cr.yp.to/djbdns/third-party.html

> RFC 2182 claims that DNS failures are not cached; that claim is false.

Here are some more recent details and the fuller explanation:

https://serverfault.com/a/824873

Note that FB.com currently expires its records in 300 seconds, which is 5 minutes.

PowerDNS (used by ordns.he.net) caches servfail for 60s by default — packetcache-servfail-ttl — which isn't very far from the 5min that you get when things aren't failing.

Personally, I do agree with DJB — I think it's a better user experience to get a DNS resolution error right away, than having to wait many minutes for the TCP timeout to occur when the host is down anyways.

Re: More details about the October 4 outage

#188

Post by adrianmonk » Tue, Oct 05, 2021, 8:36 PM UTC

Ansil849 wrote: post #73

educationcto wrote: post #68

Earlier quoted context omitted.

The original error was the network command, but the slower response and lengthy outage was partially due to the physical security they put in place to prevent malicious activity. Any event like this has multiple root causes.

Yes, but the fact that the blogpost concludes on this relatively tangential note (which notably also conveniently allows Facebook to brag about their security measures) and not on the note that their audit code was apparently itself not sufficiently audited, is what makes this deceptive spin.

No, they just wanted to cover both "what caused it?" and "why did it take too long to fix it?" since both are topics people were obviously extremely interested in.

It would have been surprising and disappointing if they didn't cover both of them.

Re: More details about the October 4 outage

#189

Post by pm2222 » Tue, Oct 05, 2021, 8:39 PM UTC

toast0 wrote: post #173

pm2222 wrote: post #152

Earlier quoted context omitted.

Ok so the DNS servers at PoPs, outside of backbone, did not go down. Does it mean they can respond with public IPs meaningful for local PoP only and are not able to respond with IPs as directions to other PoPs or FB's main DCs? So that has to mean different public IPs are handed out at different PoPs, right?

I'm not quite sure I understand the question exactly, but let me give it a try. So, first off, each pop has a /24, so like the seattle-1 pop which is near me has 157.240.3.X addresses; for me, whatsapp.net currently resolves to 157.240.3.54 in the seattle-1 pop. these addresses are used as unicast meaning they go to one place only, and they're dedicated for seattle-1 (until FB moves them around). But there are also a…

  Ok, so then I think your question is, if my DNS request for whatsapp.net makes it to the seattle1 PoP, will it only respond with a seattle-1 IP? That's one way to do it, but it's not necessarily the best way. Since my DNS requests could make it to any PoP, sending back an answer that points at that PoP may not be the best place to send me.

  Ideally, you want to send back an answer that is network local to the requester and also not a PoP that is overloaded. Every fancy DNS server does it a little different, but more or less you're integrating a bunch of information that links resolver IP to network location as well as capacity information and doing the best you can. Sometimes that would be sending users to anycast which should end up network local (but doesn't always), sometimes it's sending them to a specific pop you think is local, sometimes it's sending them to another pop because the usual best pop has some issue (overloaded on CPU, network congestion to the datacenters, network congestion on peering/transit, utility power issue, incoming weather event, fiber cut or upcoming fiber maintenance, etc).

Right I was hoping the DNSs of FB ought to be smarter than usual and let's say when DNS at Seattle-1 cannot reach backbone it'd respond with IP of perhaps NYC/SF before it starts the BGP withdrawal.

Thanks for the write up and I enjoy it.

Re: More details about the October 4 outage

#190

Post by tehjoker » Tue, Oct 05, 2021, 8:40 PM UTC

Hokusai wrote: post #121

gist wrote: post #99

Earlier quoted context omitted.

> What is the target audience of this post? Separate point to your question. FB is under no obligation to provide more details than they need to because a small segment of the population (certainly relative to their 'customers') might find it interesting or helpful or entertaining. FB is a business. They can essentially do (and should be able to) do whatever they want. There is no requirement (and there should be no…

The air industry has this solved, it's mandatory to report certain kind of incidents to avoid them in the future and inform the aviation community. https://www.skybrary.aero/index.php/Mandatory_Occurrence_Rep... That the main form of personal communication for hundreds of millions of users is down and there is no mandatory reporting is irresponsible. That Facebook is a business does not mean that they do not have res…

One would imagine a large local supermarket going down would owe the people it serves some explanation. That's where their food comes from.

At this point, I am completely sick of the pro-corporate rhetoric to let businesses do whatever they want. They exist to serve the public and they should be treated as such.