It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.
Update about the October 4th outage
21–30 of 239 posts
Re: Update about the October 4th outage
#22Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…
Without in depth technical details of what exactly happened and how, it’s hard to say. With what we have heard and seen the answer is probably that there are some opportunities to build in more resiliency, but ultimately the kind of failure that causes domino effects is impossible to eliminate completely.
This is partially due to design limitations of BGP, and partially due to it being nearly impossible to eliminate all sources of large scale failures in any highly complex system, and increasing the uptime of a system that already has a few nines costs an additional order of magnitude for each new nine. At some point you set your risk tolerance and have catastrophic failures now and then.
Re: Update about the October 4th outage
#23Reading this statement all I can think of is this scene https://www.youtube.com/watch?v=15HTd4Um1m4
Re: Update about the October 4th outage
#24Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…
If you want 1 facebook.com entity (that in turn controls instagram, whatsapp,etc) , then I get why a single AS change would take out globally. The anti-anti-pattern would be to have country specific AS's, kind of like a franchised restaurant pattern, where each country's version of fb owns their own AS that loosely forms back into the facebook mothership org, but from infrastructure point of view points to their own…
Don't AS "things" only affect the IP addresses? Surely you could have DNS records for facebook.com pointing to IP addresses in multiple ASes?
Re: Update about the October 4th outage
#25Re: Update about the October 4th outage
#26Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…
Ultimately I think you are right, but I don't think this is the right way to think about the question. It's not as though they thought they were creating a monolith on purpose by following a pattern for creating monoliths. They thought they were following best practices and building a distributed system, without single-points of failure. I think a more productive way to think about it might be what's the psychology and practices that led them to be wrong? My best guess is that at some points in the stack DNS is opaque, and at other points, particularly for trained network people, it becomes transparent (i.e. invisible) and disappears (like a mirage), so then they make both the network and physical locks dependent on it (and BGP) and... lock themselves out when it fails.
Re: Update about the October 4th outage
#27> configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues
This could be anything, potentially.
I'm not very knowledgeable in computer networking, but this could be as trivial as an incorrect update to a DNS record, right?
Re: Update about the October 4th outage
#28One of the things they restored was annoying sounds in the app every time I tap anything. Who knew that was DNS related!
Re: Update about the October 4th outage
#29The best course of action is to split FB into separate companies. It is already neatly divided between instagram, WU and legacy facebook. That would be the best for the government to avoid disruptions.
Re: Update about the October 4th outage
#30It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.
Like what?