Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

71–80 of 239 posts

Re: Update about the October 4th outage

#71
post #21

It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.

I work in a different social media company that has had some visible outages. Its always hilarious to see how wrong people are with their confident speculation. It's a good reminder that people online are often full of shit.

Re: Update about the October 4th outage

#72
post #41

Earlier quoted context omitted.

I didn’t see any disinformation, just initial reports that it was DNS which were later explained to be caused by BGP.

- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…

Most of these started out as speculation and jokes. Maybe we have a stupidity (i.e. critical thinking) problem rather than a misinformation problem. Only two of these (angle grinders and access to the offices) can really be considered lies or mistakes by an otherwise reputable source (the NYT) and one of those doesn't seem to have been refuted or retracted yet.

Re: Update about the October 4th outage

#73

Earlier quoted context omitted.

We almost went down the ‘this is a subterfuge to delete whistleblower evidence’ rabbit hole.

Did we, really?

There are a lot of comments promoting the idea here: https://news.ycombinator.com/item?id=28751224

Some of them seem to actually say it's likely, not just possible.

Re: Update about the October 4th outage

#75
post #41

Earlier quoted context omitted.

- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…

The angle grinders stuff was confirmed by NYTimes. Everything else was rank speculation by people who admitted as such. No one was passing off a hostile attack as fact.

I think you mean repeated uncritically by NYT. It doesn't even make sense, as reported. Yes, maybe to get into the cage in a shared DC, but not individual servers, and it turns out neither was necessary. That's a rehash of an old Google story.

Another bit of misreporting (e.g. The Verge) is that this was "one of its main US data centers in California" when there's no such thing. The seriously big heaps of hardware are elsewhere and they're not shared, so there are no cages to force entry into. I've been in one; no cages in sight. I know which facility they're talking about, and its only distinguishing characteristic is that it's close to where relevant people live.

Re: Update about the October 4th outage

#76
post #61

Earlier quoted context omitted.

Understood. I had question: This is what we can only perceive through internet/routing table entries right? Internal to FB, we don't know what had caused issues that led to the BGP UPDATE. That's kind of what has been confusing me - there's a lot of speculation around FB's data center design and what actually happened, but we actually don't know for sure until they post an RCA - please correct me if I'm wrong here.

You are on the right path. It's entirely possible the BGP routing updates that we witnessed 'out here' on the Internet were simply a side effect of the actual network problem that impacted the company. As toast0 mentioned above it is less likely to be DNS-related just because most of the related services don't tend to build dependencies on DNS (in part for this reason).

[deleted]

Re: Update about the October 4th outage

#77

> We also have no evidence that user data was compromised as a result of this downtime. I am not sure why they had to mention this specifically. This makes it sound like an external attack.

There were rumors early in the downtime that it was the responsibility of various outside groups. Saying, "no, your data was not impacted" is pretty standard in light of those rumors, even if they weren't the main ones spreading around after the initial reports.

Re: Update about the October 4th outage

#78

Earlier quoted context omitted.

Surely “anti-pattern” doesn’t just mean “anything with negative outcomes.” Couldn’t this just be a really big mistake that isn’t indicative of an anti-pattern?

Single point of failure is the oldest anti-pattern in the book. Goes back 5000 years.

Is there evidence to suggest that redundancy could have solved this problem? If this was simply a bad configuration that was propagated (as intended) to a large number of systems, I would hardly call that a single point of failure.

Re: Update about the October 4th outage

#79

Earlier quoted context omitted.

The angle grinders stuff was confirmed by NYTimes. Everything else was rank speculation by people who admitted as such. No one was passing off a hostile attack as fact.

And then retracted by NYT: > Correction: Oct. 4, 2021. An earlier version of this article misstated a Facebook team’s means of getting access to server computers at a data center in Santa Clara, Calif. The team did not have to cut through a cage using an industrial angle grinder.

Even if it wasn't true, drilling out locks or cutting them with tools is a pretty standard way of dealing with a lock you can't open. In an emergency where lots of things were broken it would be a perfectly reasonable thing to do. Also a news source with a nontechnical reporter might say the wrong thing that would indicate somebody was opening a server chassis with a powertool.

Re: Update about the October 4th outage

#80
post #41

Earlier quoted context omitted.

- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…

Most of these started out as speculation and jokes. Maybe we have a stupidity (i.e. critical thinking) problem rather than a misinformation problem. Only two of these (angle grinders and access to the offices) can really be considered lies or mistakes by an otherwise reputable source (the NYT) and one of those doesn't seem to have been refuted or retracted yet.

It looks like they retracted/corrected the angle grinder thing but they didn't use a retraction yet for the "access to office" info, maybe that means that was correct? (for a section of the FB workforce, of course).
Post reply on HN