Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

Post reply on HN
Print view View original

191–200 of 239 posts

Re: Update about the October 4th outage

#191

Post by dTal »

tinus_hn wrote: post #176
dTal wrote: post #161

Earlier quoted context omitted.

On the off-chance this isn't sarcasm, Facebook's routing shenanigans slowed down the entire internet . Not to mention that they're a publicly traded company, and one which has gone out of its way to assume an infrastructure role. They don't have a right to privacy here, and we are all owed an explanation.

Or else? You will angrily stamp your foot? Start an e-petition?

This comment thread is about whether we ought to receive an explanation, not the practical likelihood of getting one.

Re: Update about the October 4th outage

#192

Post by dTal »

ikt wrote: post #163
dTal wrote: post #161

Earlier quoted context omitted.

On the off-chance this isn't sarcasm, Facebook's routing shenanigans slowed down the entire internet . Not to mention that they're a publicly traded company, and one which has gone out of its way to assume an infrastructure role. They don't have a right to privacy here, and we are all owed an explanation.

I don't want an explanation nor do I care, Facebook could disappear tomorrow like all the other networks before it and it wouldn't make a dent in my day.

Okay? Why are you commenting here then?

Re: Update about the October 4th outage

#194

Post by aeyes »

jameshart wrote: post #190
laegooose wrote: post #105

Earlier quoted context omitted.

How would you explain that AWS, GCE, Cloudflare, GitLab publish very detailed post-mortems?

Are you under the impression that Facebook is a SaaS provider? Facebook sells ad space, retail. The impact on their customers of the outage is ‘sorry, you couldn’t buy ads for a few hours.’ Demanding a public RCA for this is like demanding an RCA from Costco because they’re out of stock of tinned beans.

WhatsApp for Business and Instagram Shops for example are SaaS offerings.

Re: Update about the October 4th outage

#195

Post by oarsinsync »

dTal wrote: post #161
tinus_hn wrote: post #141

Earlier quoted context omitted.

It’s not reasonable to demand any details at all, it’s nice of them to notify people of what went wrong but it really is none of our business.

On the off-chance this isn't sarcasm, Facebook's routing shenanigans slowed down the entire internet . Not to mention that they're a publicly traded company, and one which has gone out of its way to assume an infrastructure role. They don't have a right to privacy here, and we are all owed an explanation.

> Facebook's routing shenanigans slowed down the entire internet

This is Hacker News, so the distinction between network performance, server performance and application performance should matter.

"The Internet" did not slow down. "The Internet" infact probably had more available capacity as a result of Facebook's outage, as all those bits of outrage and cats ceased to be transferred for the duration.

Some applications may have seen performance hits, as a result of poorly thought out dependencies on an external service without graceful failure.

Some applications may have seen increased load and suffered due to server resourcing constraints, caused by applications like the above failing to fail gracefully, and instead polling more aggressively.

> They don't have a right to privacy here, and we are all owed an explanation

Morally / ethically, you're right. The fact that Facebook exists in it's current form tells me that morals and ethics aren't particularly important to the real world.

Re: Update about the October 4th outage

#197

Post by dTal »

oarsinsync wrote: post #195
dTal wrote: post #161

Earlier quoted context omitted.

On the off-chance this isn't sarcasm, Facebook's routing shenanigans slowed down the entire internet . Not to mention that they're a publicly traded company, and one which has gone out of its way to assume an infrastructure role. They don't have a right to privacy here, and we are all owed an explanation.

> Facebook's routing shenanigans slowed down the entire internet This is Hacker News, so the distinction between network performance, server performance and application performance should matter. "The Internet" did not slow down. "The Internet" infact probably had more available capacity as a result of Facebook's outage, as all those bits of outrage and cats ceased to be transferred for the duration. Some application…

Perhaps you're unaware that billions of devices attempting to resolve Facebook's unresolvable domains effectively DDOS-ed the DNS system? It most certainly did slow down big chunks of internet which otherwise had nothing to do with Facebook.

https://www.theverge.com/2021/10/4/22709123/facebook-outage-...

Re: Update about the October 4th outage

#198

Post by marcosfelt »

reilly3000 wrote: post #167

So their actual deployment process is quite rigorous and should have a tight blast radius. After lots of emulated and canary testing, their deployments are phased out over weeks. I don't see how a bad push could have done what happened yesterday. I found a paper that describes the process in detail. See page 10-11: https://web.archive.org/web/20211005034928/https://research.... Phase Specification P1 Small number of…

If they have such a rigorous release process, what could have caused all of the dns records to get wiped?

Re: Update about the October 4th outage

#199

Post by quartz »

go_prodev wrote: post #162

I worked with a network engineer who misconfigured a router that was connecting a bank to it's DR site. The engineer had to drive across town to manually patch into the router to fix it. DR downtime was about an hour, but the bank fired him anyway. Given that Zuck lost a substantial amount of money, I wonder if the engineer faced any ramifications. Sidenote: I asked the bank infrastructure team why the DR site was in…

Facebook has a very healthy approach to incident response (one of the reasons it's so rare for the site to go down at all despite the enormous traffic and daily code pushes).

Unless there was some kind of nefarious intent, it's very unlikely anyone will be 'punished'. The likely ramifications will be around changes to processes, tests, automations, and fallbacks to 1) prevent the root sequence of events from happening again and 2) make it easier to recover from similar classes of problems in the future.

Re: Update about the October 4th outage

#200

Post by ethbr0 »

quartz wrote: post #199
go_prodev wrote: post #162

I worked with a network engineer who misconfigured a router that was connecting a bank to it's DR site. The engineer had to drive across town to manually patch into the router to fix it. DR downtime was about an hour, but the bank fired him anyway. Given that Zuck lost a substantial amount of money, I wonder if the engineer faced any ramifications. Sidenote: I asked the bank infrastructure team why the DR site was in…

Facebook has a very healthy approach to incident response (one of the reasons it's so rare for the site to go down at all despite the enormous traffic and daily code pushes). Unless there was some kind of nefarious intent, it's very unlikely anyone will be 'punished'. The likely ramifications will be around changes to processes, tests, automations, and fallbacks to 1) prevent the root sequence of events from happenin…

I've never understood companies that fire individuals when policies were followed and an incident happened. Or, when no policies existed. Or, when policies are routinely bypassed.

Organizational failures require organizational solutions. That seems pretty obvious.

Post reply on HN