Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

81–90 of 306 posts

Re: More details about the October 4 outage

#81
post #48

> Our primary and out-of-band network access was down Don't create circular dependencies.

With something as fundamental as the network, no way around it. - Okay, we'll set up a separate maintenance network in case we can't get to the regular network. - Wait, but we need a maintenance network for the maintenance network...

Two is One, One is None. There are absolutely ways around this, it's called redundancy. The marginal cost of laying an extra pair during physical plant installation is basically $0, which is why you'd never go "well we need a backup for the backup, so there's no point in having two pairs). Similarly, the marginal cost for having a second UPS and PDU in a rack is effectively $0 at scale, so nobody would argue this is unnecessary to deal with possible UPS failure or accidentally unplugging a cable.

In this case, there are likely several things that can be changes systemically to mitigate or prevent similar failures in the future, and I have every faith that Facebook's SRE team is capable of identifying and implementing those changes. There is no such thing as "no way around it", unless you're dealing with a law of physics.

Re: More details about the October 4 outage

#82
post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> I'd be curious to see what systemic changes happen at FB as a result, if any.

If history is any guide, Facebook will decide some division charged with preventing problems was an ineffective waste of money, shut it down, and fire a bunch of people.

Re: More details about the October 4 outage

#83

> We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this. From here on out, our job…

Simple Testing Can Prevent Most Critical Failures[1], "We found the majority of catastrophic failures could easily have been prevented by performing simple testing on error handling code – the last line of defense – even without an understanding of the software design." 1 https://www.eecg.utoronto.ca/~yuan/papers/failure_analysis_o...

That article should be required reading for all of us.

Re: More details about the October 4 outage

#84
post #39

Apparently they had to bring in the angle grinder to get access to the server room. https://twitter.com/cullend/status/1445156376934862848?t=P5u...

Was this ever confirmed? NYT tech reporter Mike Isaac issued a correction to his previous reporting about it. > the team dispatched to the Facebook site had issues getting in because of physical security but did not need to use a saw/ grinder. https://twitter.com/MikeIsaac/status/1445196576956162050

> so.....they had to use a jackhammer, got it

Re: More details about the October 4 outage

#85

It is completely logical but still kind of amazing, that facebook plugged their globally distributed datacenters together with physical wire.

What do you imagine other companies use to connect their datacenters?

Uh... the cloud?

Re: More details about the October 4 outage

#87

Apparently they had to bring in the angle grinder to get access to the server room. https://twitter.com/cullend/status/1445156376934862848?t=P5u...

or not

https://twitter.com/MikeIsaac/status/1445196576956162050

https://twitter.com/cullend/status/1445212476652535815

Re: More details about the October 4 outage

#88

Earlier quoted context omitted.

Going on at lengths about how the trade off between prolonged downtime and strict security protocols is a worthy trade off is erecting a nonsensical strawman, the literal definition of spinning a story. The key issue had nothing to do with Facebook's data center security protocols.

No one except you is erecting a nonsensical strawman.

Except, I quoted exactly where they were.

Re: More details about the October 4 outage

#89

> Our primary and out-of-band network access was down Don't create circular dependencies.

How do you avoid circular dependencies on an out-of-band-network? Seems like the choice is between a circular dependency, or turtles all the way down.

How do you go from "have a separate access method that doesn't depend on your main system" to "turtles all the way down"? The secondary access is allowed to have dependencies, just not on your network.

Re: More details about the October 4 outage

#90
post #34

Earlier quoted context omitted.

To expand on why this made me think of the Google outage: It was a global backbone isolation, caused by configuration changes (as they all are...). It was detected fairly early on, but recovery was difficult because internal tools / debugging workflows were also impacted, and even after the problem was identified, it still took time to back out the change. "But wait, a global backbone isolation? Google wasn't totally…

But the FB outage was not a configuration change. > a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network

This is kind of like Chernobyl where they were testing to see how hot they could run the reactor to see how much power it could generate. Then things went sideways.
Post reply on HN