Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

21–30 of 306 posts

Re: More details about the October 4 outage

#21
post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

The media. This was a huge international story.

Re: More details about the October 4 outage

#23
post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

Both those groups of people. I imagine, they would either be accused of it being either too complicated or dumbed down, so they do both in the same article.

Re: More details about the October 4 outage

#24

> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally Imagine being this person. Tomorrow on /r/tifu.

of course if one person can knock down an entire global system through a trivial mistake the problem is obviously not the person to begin with, but the architecture of the system.

Re: More details about the October 4 outage

#25
post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

> Those data centers come in different forms.

It's like the birds and the bees.

Re: More details about the October 4 outage

#26
post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

Facebook also had a nearly 24-hour outage in 2019. https://www.nytimes.com/2019/03/14/technology/facebook-whats... (or http://archive.today/O7ycB )

Re: More details about the October 4 outage

#28

> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally Imagine being this person. Tomorrow on /r/tifu.

I doubt Facebook engineers are free-typing commands on Bash, so it’s probably not an individual error. More likely to be a race condition or other edge case that wasn’t considered during a review. This might be a script that’s run 1000s of times before with no problems.

Re: More details about the October 4 outage

#30
Was this a way to delete a lot of evidence before shit really hit the fan?'

After reading this, I can't help but feel this was a calculated move.

It gives FB a chance to hijack media attention from the whistleblower. It gives them a chance to show the average peson, 'hey, we make mistakes and we have a review process to improve our systems'.

The timing is too perfect if you ask me.

Post reply on HN