It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.
Update about the October 4th outage
71–80 of 239 posts
Re: Update about the October 4th outage
#72Earlier quoted context omitted.
I didn’t see any disinformation, just initial reports that it was DNS which were later explained to be caused by BGP.
- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…
Re: Update about the October 4th outage
#73Earlier quoted context omitted.
We almost went down the ‘this is a subterfuge to delete whistleblower evidence’ rabbit hole.
Did we, really?
Some of them seem to actually say it's likely, not just possible.
Re: Update about the October 4th outage
#74I am not sure why they had to mention this specifically. This makes it sound like an external attack.
Re: Update about the October 4th outage
#75Earlier quoted context omitted.
- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…
The angle grinders stuff was confirmed by NYTimes. Everything else was rank speculation by people who admitted as such. No one was passing off a hostile attack as fact.
Another bit of misreporting (e.g. The Verge) is that this was "one of its main US data centers in California" when there's no such thing. The seriously big heaps of hardware are elsewhere and they're not shared, so there are no cages to force entry into. I've been in one; no cages in sight. I know which facility they're talking about, and its only distinguishing characteristic is that it's close to where relevant people live.
Re: Update about the October 4th outage
#76Earlier quoted context omitted.
Understood. I had question: This is what we can only perceive through internet/routing table entries right? Internal to FB, we don't know what had caused issues that led to the BGP UPDATE. That's kind of what has been confusing me - there's a lot of speculation around FB's data center design and what actually happened, but we actually don't know for sure until they post an RCA - please correct me if I'm wrong here.
You are on the right path. It's entirely possible the BGP routing updates that we witnessed 'out here' on the Internet were simply a side effect of the actual network problem that impacted the company. As toast0 mentioned above it is less likely to be DNS-related just because most of the related services don't tend to build dependencies on DNS (in part for this reason).
Re: Update about the October 4th outage
#77> We also have no evidence that user data was compromised as a result of this downtime. I am not sure why they had to mention this specifically. This makes it sound like an external attack.
Re: Update about the October 4th outage
#78Earlier quoted context omitted.
Surely “anti-pattern” doesn’t just mean “anything with negative outcomes.” Couldn’t this just be a really big mistake that isn’t indicative of an anti-pattern?
Single point of failure is the oldest anti-pattern in the book. Goes back 5000 years.
Re: Update about the October 4th outage
#79Earlier quoted context omitted.
The angle grinders stuff was confirmed by NYTimes. Everything else was rank speculation by people who admitted as such. No one was passing off a hostile attack as fact.
And then retracted by NYT: > Correction: Oct. 4, 2021. An earlier version of this article misstated a Facebook team’s means of getting access to server computers at a data center in Santa Clara, Calif. The team did not have to cut through a cage using an industrial angle grinder.
Re: Update about the October 4th outage
#80Earlier quoted context omitted.
- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…
Most of these started out as speculation and jokes. Maybe we have a stupidity (i.e. critical thinking) problem rather than a misinformation problem. Only two of these (angle grinders and access to the offices) can really be considered lies or mistakes by an otherwise reputable source (the NYT) and one of those doesn't seem to have been refuted or retracted yet.