Update about the October 4th outage
engineering.fb.com
Update about the October 4th outage
1–10 of 239 posts
Re: Update about the October 4th outage
#2The first thing people here thought of was that it was the gouvernement denying access to these websites as it usually does for a number of reasons.
Re: Update about the October 4th outage
#3It has been painfully admitted by the Facebook mafia that they know that they are the internet and farming the data of an entire civilisation; further evidence that this deep integration of their services needs to be broken up.
After all the scandals, leaks, whistleblowers etc it would take more than a DNS record wipe to take down the Facebook mafia.
Re: Update about the October 4th outage
#4It was interesting to visit the subreddits of random countries (eg /r/Mongolia) and see the top posts all asking if fb/Insta/WhatsApp being down was local or global. I got the impression this morning that it was only affecting NA and Europe, but it looks like it was totally global. The numbers must be staggering of the number of people trying to login.
Re: Update about the October 4th outage
#5Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower?
By having a large centralized and monolithic system, aren't they guaranteeing that mistakes cause huge splash damage and don't separate concerns?
Re: Update about the October 4th outage
#6The first thing people here thought of was that it was the gouvernement denying access to these websites as it usually does for a number of reasons.
It was pretty quickly deemed a global phenomenon, so no comments on posts about it said that. Also, enough people here on HN know how to investigate dns and bgp to have found the problem within the first 30 minutes, first with DNS then the revelation that every BGP route associated with them was withdrawn.
Re: Update about the October 4th outage
#7This is a funny post to have suggested at the bottom of the article: https://engineering.fb.com/2021/08/09/connectivity/backbone-...
Re: Update about the October 4th outage
#8This more or less confirms what we’ve heard, and I appreciate the speed, but it’s incredibly lame from a details point of view.
Will a real postmortem follow? Or is this the best we are gonna get?
Re: Update about the October 4th outage
#9Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…
One might argue that the fact that you care about things like this is why you don’t run Facebook.
Re: Update about the October 4th outage
#10Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…
I mean if you own 3 independent businesses, each of which would be worth over $100 billion, and you break all of them simultaneously for an entire day, including your internal email and your badge entry systems, yes, that is definitionally an anti-pattern.