Update about the October 4th outage • Hacker News Board

Re: Update about the October 4th outage

#101

Post by bawolff » Tue, Oct 05, 2021, 6:22 AM UTC

padolsey wrote: post #81

johnduhart wrote: post #35

Earlier quoted context omitted.

I think you need to re-adjust your expectations, it's not reasonable to have a fully fleshed out RCA blog post available within hours of incident resolution. Most other cloud providers take a few days for theirs.

I mean, not an RCA per se, but info more akin to cloudflare's blog post would be v welcome IMHO: https://blog.cloudflare.com/october-2021-facebook-outage/

Both posts have essentially the same info - the fb one just didn't include an explainer on how the internet works.

Re: Update about the October 4th outage

#102

Post by toast0 » Tue, Oct 05, 2021, 6:22 AM UTC

advpetc wrote: post #66

jcims wrote: post #57

Earlier quoted context omitted.

It was also down during the outage.

That’s a bit sad

For a status page to be actually independent, it needs to have all it's requirements hosted on other infrastructure. fb.com authoritative DNS is the same as facebook.com, so it's going down when (FB) DNS goes down (and DNS is going down when BGP is broken, apparently).

It looks like the status page is hosted on CloudFront though, so it got part of the way. (Of course, the other question is if it was updatable / updated during the outage)

Re: Update about the October 4th outage

#103

Post by bawolff » Tue, Oct 05, 2021, 6:25 AM UTC

tomrod wrote: post #30

paxys wrote: post #21

It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.

Like what?

Lots of people blaming dns

Re: Update about the October 4th outage

#104

Post by andy-x » Tue, Oct 05, 2021, 6:25 AM UTC

Such a BS. FB imagining that they are their own Internet but failing in a most miserable way because they need actual Internet to communicate.

Re: Update about the October 4th outage

#105

Post by laegooose » Tue, Oct 05, 2021, 6:25 AM UTC

badtux wrote: post #91

andrewxdiamond wrote: post #8

This more or less confirms what we’ve heard, and I appreciate the speed, but it’s incredibly lame from a details point of view. Will a real postmortem follow? Or is this the best we are gonna get?

Having been on the team that issued postmortems before, I can tell you that we said as little as possible in as vague a way as possible while meeting our minimum legal requirements. Actual Facebook customers (i.e. those who pay money to Facebook) will get a slightly more detailed release. But the whole goal is to give as little information as possible while appearing to be open. As an engineer that makes me growl, bu…

How would you explain that AWS, GCE, Cloudflare, GitLab publish very detailed post-mortems?

Re: Update about the October 4th outage

#106

Post by toast0 » Tue, Oct 05, 2021, 6:26 AM UTC

manquer wrote: post #18

0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

BGP has to converge to a single routing table. You are effectively asking is why is there a single routing table for the internet. To put in simple terms having a single routing table is what it makes it the internet we can share, otherwise it would just be a bunch of independent networks.

> BGP has to converge to a single routing table.

It certainly does not. If I peer with you, neither of us (generally) announce that route to our other peers, but often announce to our customers. There are many routes that are not visible to everyone, and there is no single routing table for the internet. Each BGP speaker ends up with their own routing table, although there are a lot of similarities.

Re: Update about the October 4th outage

#107

Post by withinboredom » Tue, Oct 05, 2021, 6:28 AM UTC

tshaddox wrote: post #78

spoonjim wrote: post #49

Earlier quoted context omitted.

Single point of failure is the oldest anti-pattern in the book. Goes back 5000 years.

Is there evidence to suggest that redundancy could have solved this problem? If this was simply a bad configuration that was propagated (as intended) to a large number of systems, I would hardly call that a single point of failure.

If it was a single change, deployed once that caused this, then it’s a single point of failure.

Re: Update about the October 4th outage

#108

Post by toast0 » Tue, Oct 05, 2021, 6:34 AM UTC

tw04 wrote: post #13

0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

Quite the opposite. Back in the day you would've had to login dozens if not hundreds of routers individually to push the change, and it likely would've been caught after screwing up the first one. This is the result of SDN (software defined networking) and being able to push a change globally from one command. I recall major ISP's screwing up their routing tables in the past but never globally on this level.

https://www.itproportal.com/news/misconfigured-centurylink-d...

https://www.bleepingcomputer.com/news/technology/ibm-cloud-g... (this one isn't clear, maybe BGP hijacking, and if so, not sure who the responsible party was)

https://www.catchpoint.com/blog/vodafone-idea-bgp-leak (not sure how major this one was)

You can practically search ISP bgp outage and get news about the last couple times they screwed up BGP and caused a big problem. Or service BGP and get a 50/50 chance of the service screwing up BGP or an ISP/country hijacking their routes and causing a big problem.

BGP is one of the best ways to break things at scale.

Re: Update about the October 4th outage

#109

Post by KronisLV » Tue, Oct 05, 2021, 6:40 AM UTC

paxys wrote: post #37

geerlingguy wrote: post #14

Gotta love how painfully vague this is. Sounds like a PR piece for investors, not an engineering blog piece.

RCAs take time. It's best to issue vague statements during and right after an incident rather than make guesses.

> It's best to issue vague statements during and right after an incident rather than make guesses.

Why? Why couldn't you just post that the RCA is still ongoing and that proper updates will follow? Otherwise all you get is meaningless fluff.

Re: Update about the October 4th outage

#110

Post by melvinmt » Tue, Oct 05, 2021, 6:40 AM UTC

imgabe wrote: post #87

It just occurred to me to wonder if Facebook has a Twitter account and if they used it to update people about the outage. It turns out they do, and they did, which makes sense. Boy, it must have been galling to have to use a competing communication network to tell people that your network is down. It looks like Zuckerberg doesn't have a personal Twitter though, nor does Jack Dorsey have a public Facebook page (or the…

> It looks like Zuckerberg doesn't have a personal Twitter though

He does: https://twitter.com/finkd