More details about the October 4 outage • Hacker News Board

Re: More details about the October 4 outage

#211

Post by dfcowell » Tue, Oct 05, 2021, 9:37 PM UTC

Johnny555 wrote: post #147

dfcowell wrote: post #127

Earlier quoted context omitted.

All of the components in the supply chain will be rated for greater than max load, however power generation at grid scale is a delicate balancing act. I’m not an electrical engineer, so the details here may be fuzzy, however in broad strokes: Grid operators constantly monitor power consumption across the grid. If more power is being drawn than generated, line frequency drops across the whole grid. This leads to brown…

You're talking about the grid, the OP was talking about datacenter infrastructure -- which one is the weak link? If a datacenter can't go from idle (but powered on) servers to fully utilized servers without taking down the power grid, then it seems that they'd have software controls in place to prevent this, since there are other failure modes that could cause this behavior other than a global Facebook outage.

Unfortunately the article doesn’t provide enough explicit detail to be 100% sure one way or the other, however my read is that it’s probably the grid.

> Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk.

“Electrical systems” is vague and could refer to either internal systems, external systems or both.

That said, if the DC is capable of running under sustained load at peak (which we have to assume it is, since that’s its normal state when FB is operational) it seems to me like the externality of the grid is the more likely candidate.

In terms of software controls preventing this kind of failure mode, they do have it - load shedding. They’ll cut your supply until capacity is made available.

Re: More details about the October 4 outage

#212

Post by mbauman » Tue, Oct 05, 2021, 9:44 PM UTC

cnst wrote: post #210

mbauman wrote: post #199

Earlier quoted context omitted.

How would've it been worse? Waiting for a timeout is a good thing as it prevents a thundering herd of refresh-smashing (both automated and manual). I don't know BGP well, but it seems easier for peers to just drop FB's packets on the floor than deal with a DNS stampede.

An average webpage today is several megabytes in size. How would a few bytes over a couple of UDP packets for DNS have any meaningful impact on anyone's network? If anything, things fail faster, so, there's less data to transmit. For example, I often use ordns.he.net as an open recursive resolver. They use PowerDNS as their software. PowerDNS has the default of packetcache-servfail-ttl of 60s. OTOH, fb.com A response…

The rest of the internet sucked yesterday, and my understanding was it was due to a thundering herd of recursive DNS requests. Slowing down clients seems like a good thing.

Re: More details about the October 4 outage

#213

Post by byron22 » Tue, Oct 05, 2021, 9:46 PM UTC

What kind of bgp command would do that?

Re: More details about the October 4 outage

#214

Post by harshreality » Tue, Oct 05, 2021, 9:50 PM UTC

jaywalk wrote: post #146

harshreality wrote: post #104

> To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. No, it's (clearly) not a guaranteed indication of that. Logic fail. Infrastructure tools at that scale need to handle all possible causes of test failures. "Is the internet down or only the few sites I'm testing?" is a cl…

I think you're misunderstanding. The DNS servers (at Facebook peering points) had zero access to Facebook datacenters because the backbone was down. That is as unhealthy as the network connection can get, so they (correctly) stopped advertising the routes to the outside world. By that point, the Facebook backbone was already gone. The DNS servers stopping BGP advertisements to the outside world did not cause that.

You're talking about backend network connections to facebook's datacenters as if that's the only thing that matters. I'm talking about overall network connection including the internet-facing part.

Facebook's infrastructure at their peering points loses all contact with their respective facebook datacenter(s).

Their response is to automatically withdraw routes to themselves. I suppose they assumed that all datacenters would never go down at the same time, so that client dns redundancy would lead to clients using other dns servers that could still contact facebook datacenters. It's unclear how those routes could be restored without on-site intervention. If they automatically detect when the datacenters are reachable again, that too requires on-site intervention since after withdrawing routes FB's ops tools can't do anything to the relevant peering points or datacenters.

But even without the catastrophic case of all datacenter connections going down, you don't need to be a facebook ops engineer to realize that there are problems that need to be carefully thought through when ops tools depends on the same (public) network routes and DNS entries that the DNS servers are capable of autonomously withdrawing.

Re: More details about the October 4 outage

#215

Post by pm2222 » Tue, Oct 05, 2021, 10:00 PM UTC

toast0 wrote: post #198

pm2222 wrote: post #189

Earlier quoted context omitted.

Ok, so then I think your question is, if my DNS request for whatsapp.net makes it to the seattle1 PoP, will it only respond with a seattle-1 IP? That's one way to do it, but it's not necessarily the best way. Since my DNS requests could make it to any PoP, sending back an answer that points at that PoP may not be the best place to send me. Ideally, you want to send back an answer that is network local to the requeste…

> Right I was hoping the DNSs of FB ought to be smarter than usual and let's say when DNS at Seattle-1 cannot reach backbone it'd respond with IP of perhaps NYC/SF before it starts the BGP withdrawal. The problem there is coordination. The PoPs don't generally communicate amongst themselves (and may not have been able to after the FB backbone was broken, although technically, they could have through transit connectiv…

You are to the point and precise. This is exactly the problem.

  Each individual PoP did the right thing by dropping out of the anycast, but because they all did it, it was the wrong thing.

Somehow I feel the design is flawed because if abuses DNS server status a bit. I mean DNS server down and BGP withdrawal for the DNS server is a perfect combination, however connectivity between DNS and backend server down, DNS up and BGP withdrawal for DNS server is not. DNS did not fail and DNS should just fall back to some other operational DNS perhaps a regional/global default one.

Re: More details about the October 4 outage

#216

Post by [deleted] » Tue, Oct 05, 2021, 10:04 PM UTC

lvs wrote: post #194

UncleMeat wrote: post #165

Earlier quoted context omitted.

Hi there. I'm a Googler and I've directly interfaced with a nontrivial number of customers such that I alone have interfaced with more than 0.000001% of the entire world population.

All you need to do is browse any online forum, bug tracker, subreddit dedicated to a consumer-facing Google product to know that Google does not give a rat's ass about customer service. We know the customer is ultimately not the consumer.

[deleted]

Re: More details about the October 4 outage

#217

Post by dbbk » Tue, Oct 05, 2021, 10:20 PM UTC

nonbirithm wrote: post #46

DNS seems to be a massive point of failure everywhere, even taking out the tools to help deal with outages themselves. The same thing happened to Azure multiple times in the past, causing complete service outages. Surely there must be some way to better mitigate DNS misconfiguration by now, given the exceptional importance of DNS?

Seems like the simplest solution would be to just move recovery tooling to their own domain / DNS?

Re: More details about the October 4 outage

#218

Post by formerly_proven » Tue, Oct 05, 2021, 10:21 PM UTC

trevorishere wrote: post #129

formerly_proven wrote: post #122

Earlier quoted context omitted.

The Chernobyl test was not a test to drive the reactor to the limits, but actually a test to verify that the inertia of the main turbines is big enough to drive the coolant pumps for X amount of time in the case of grid failure.

Of possible interest: https://www.youtube.com/watch?v=Ijst4g5KFN0 This is a presentation to students by an MIT professor that goes over exactly what happened, the sequence of events, mistakes made, and so on.

Warning for others: I watched the above video and then watched the entire course (>30 hours).

Re: More details about the October 4 outage

#219

Post by ecnahc515 » Tue, Oct 05, 2021, 11:02 PM UTC

packetslave wrote: post #51

blowski wrote: post #28

Earlier quoted context omitted.

I doubt Facebook engineers are free-typing commands on Bash, so it’s probably not an individual error. More likely to be a race condition or other edge case that wasn’t considered during a review. This might be a script that’s run 1000s of times before with no problems.

Back in Ye Old Dark Ages, I caused a BIG Google outage by running a routine maintenance script that had been run dozens if not hundreds of times before. Turns out the underlying network software had a race condition that would ONLY be hit if the script ran at the exact same time as some automated monitoring tools polled the box. At FAANG scale, "one in a million" happens a lot more often than you'd think.

> At FAANG scale, "one in a million" happens a lot more often than you'd think.

And it happens less than you think too, sometimes, which I think is closer to the original point.

Re: More details about the October 4 outage

#220

Post by mostdataisnice » Tue, Oct 05, 2021, 11:44 PM UTC

nuerow wrote: post #142

mumblemumble wrote: post #132

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

> I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error. Wouldn't human error reflect extremely poorly on the company though? I mean, for human error to be the root cause of this mega-outage, that would imply that the company's infrastructure and operational and security practices were so ineffective that a single person scre…

I mean, sure? mumblemumble is still right though. If you're looking for a cynical reason for everything FB related, then, sure, it's true that a human error looks bad.