Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

Post reply on HN
Print view View original

11–20 of 239 posts

Re: Update about the October 4th outage

#11

Post by barkingcat »

0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

If you want 1 facebook.com entity (that in turn controls instagram, whatsapp,etc) , then I get why a single AS change would take out globally.

The anti-anti-pattern would be to have country specific AS's, kind of like a franchised restaurant pattern, where each country's version of fb owns their own AS that loosely forms back into the facebook mothership org, but from infrastructure point of view points to their own set of AS netspace.

Re: Update about the October 4th outage

#13

Post by tw04 »

0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

Quite the opposite. Back in the day you would've had to login dozens if not hundreds of routers individually to push the change, and it likely would've been caught after screwing up the first one. This is the result of SDN (software defined networking) and being able to push a change globally from one command.

I recall major ISP's screwing up their routing tables in the past but never globally on this level.

Re: Update about the October 4th outage

#15

Post by andrewxdiamond »

0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

You are correct, but this problem isn’t Facebook’s doing. This is just how BGP works. Even big players like Verizon can screw it up and break the Internet.

For all it’s flaws, BGP is the piece of the Internet that truly makes it decentralized. Without it, there would be a centralized routing table of some sort.

Re: Update about the October 4th outage

#16

Post by Jugurtha »

judge2020 wrote: post #6
Jugurtha wrote: post #2

The first thing people here thought of was that it was the gouvernement denying access to these websites as it usually does for a number of reasons.

It was pretty quickly deemed a global phenomenon, so no comments on posts about it said that. Also, enough people here on HN know how to investigate dns and bgp to have found the problem within the first 30 minutes, first with DNS then the revelation that every BGP route associated with them was withdrawn.

I have been unclear. By "here", I meant the country I am in.

Re: Update about the October 4th outage

#18

Post by manquer »

0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

BGP has to converge to a single routing table.

You are effectively asking is why is there a single routing table for the internet.

To put in simple terms having a single routing table is what it makes it the internet we can share, otherwise it would just be a bunch of independent networks.

Re: Update about the October 4th outage

#19

Post by judge2020 »

Jugurtha wrote: post #16
judge2020 wrote: post #6

Earlier quoted context omitted.

It was pretty quickly deemed a global phenomenon, so no comments on posts about it said that. Also, enough people here on HN know how to investigate dns and bgp to have found the problem within the first 30 minutes, first with DNS then the revelation that every BGP route associated with them was withdrawn.

I have been unclear. By "here", I meant the country I am in.

Sorry, thought you meant HN.

Re: Update about the October 4th outage

#20

Post by wmf »

spoonjim wrote: post #10
0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

I mean if you own 3 independent businesses, each of which would be worth over $100 billion, and you break all of them simultaneously for an entire day, including your internal email and your badge entry systems, yes, that is definitionally an anti-pattern.

One of the usual justifications for acquisitions is to save money using common infrastructure. Instagram and WhatsApp haven't been independent for a while.
Post reply on HN