Update about the October 4th outage

Post by jcims » Tue, Oct 05, 2021, 3:53 AM UTC

shahsyed wrote: post #55

dodobirdlord wrote: post #40

Earlier quoted context omitted.

There seems to be nothing uncertain about the immediate cause of the issue - Facebook revoked all of their BGP routes, and all of their IP addresses couldn't receive packets until they were restored.

Understood. I had question: This is what we can only perceive through internet/routing table entries right? Internal to FB, we don't know what had caused issues that led to the BGP UPDATE. That's kind of what has been confusing me - there's a lot of speculation around FB's data center design and what actually happened, but we actually don't know for sure until they post an RCA - please correct me if I'm wrong here.

Post by deadbolt » Tue, Oct 05, 2021, 3:58 AM UTC

runawaybottle wrote: post #33

tomrod wrote: post #30

Earlier quoted context omitted.

Like what?

We almost went down the ‘this is a subterfuge to delete whistleblower evidence’ rabbit hole.

Post by kanbara » Tue, Oct 05, 2021, 4:08 AM UTC

tw04 wrote: post #13

0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

Quite the opposite. Back in the day you would've had to login dozens if not hundreds of routers individually to push the change, and it likely would've been caught after screwing up the first one. This is the result of SDN (software defined networking) and being able to push a change globally from one command. I recall major ISP's screwing up their routing tables in the past but never globally on this level.

Post by carabiner » Tue, Oct 05, 2021, 4:41 AM UTC

paxys wrote: post #41

colechristensen wrote: post #34

Earlier quoted context omitted.

I didn’t see any disinformation, just initial reports that it was DNS which were later explained to be caused by BGP.

- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…

Post by Thorrez » Tue, Oct 05, 2021, 4:52 AM UTC

colechristensen wrote: post #44

paxys wrote: post #41

Earlier quoted context omitted.

- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…

I entirely believed that the badge readers on doors were broken, I have set systems like that up and they’re network connected. It is not the least bit surprising that BGP chaos would break that. That doesn’t mean people literally couldn’t get in the building, more like security was really annoyed and confused all day.