Gotta love how painfully vague this is. Sounds like a PR piece for investors, not an engineering blog piece.
I think you need to re-adjust your expectations, it's not reasonable to have a fully fleshed out RCA blog post available within hours of incident resolution. Most other cloud providers take a few days for theirs.
Update about the October 4th outage
81–90 of 239 posts
Re: Update about the October 4th outage
#82Well that happened already. No worries.
Re: Update about the October 4th outage
#83Earlier quoted context omitted.
I didn’t see any disinformation, just initial reports that it was DNS which were later explained to be caused by BGP.
- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…
Re: Update about the October 4th outage
#84> We also have no evidence that user data was compromised as a result of this downtime. I am not sure why they had to mention this specifically. This makes it sound like an external attack.
Re: Update about the October 4th outage
#85It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.
I work in a different social media company that has had some visible outages. Its always hilarious to see how wrong people are with their confident speculation. It's a good reminder that people online are often full of shit.
It amazing how wrong people can be and how confident they are about being right.
Even sometimes fighting _me_ about things _I_ designed and built.
It’s quite sobering; taught me not to believe all the speculation I read.
Re: Update about the October 4th outage
#86The badge system should be local to the building. There are few actual reasons (sure, besides "efficiency") of why badge control should be centralized. Even less reasons for it to be a subdomain of fb. Another option would be to keep the system but make it failsafe (but it seems the newer generation doesn't know what that means). If the network goes down keep it at the last config. Badge validation should be offline first and added/removed ones should be broadcast periodically.
This is the same issue with smartlocks times the number of employees. Do you really want to add another point of failure between yourself and your home?
Re: Update about the October 4th outage
#87It looks like Zuckerberg doesn't have a personal Twitter though, nor does Jack Dorsey have a public Facebook page (or they're set not to show up in search).
Re: Update about the October 4th outage
#88Earlier quoted context omitted.
I work in a different social media company that has had some visible outages. Its always hilarious to see how wrong people are with their confident speculation. It's a good reminder that people online are often full of shit.
I work in video games. It amazing how wrong people can be and how confident they are about being right. Even sometimes fighting _me_ about things _I_ designed and built. It’s quite sobering; taught me not to believe all the speculation I read.
Re: Update about the October 4th outage
#89Re: Update about the October 4th outage
#90It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.