Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

Post reply on HN
Print view View original

111–120 of 239 posts

Re: Update about the October 4th outage

#111

Post by _nalply »

laegooose wrote: post #105
badtux wrote: post #91

Earlier quoted context omitted.

Having been on the team that issued postmortems before, I can tell you that we said as little as possible in as vague a way as possible while meeting our minimum legal requirements. Actual Facebook customers (i.e. those who pay money to Facebook) will get a slightly more detailed release. But the whole goal is to give as little information as possible while appearing to be open. As an engineer that makes me growl, bu…

How would you explain that AWS, GCE, Cloudflare, GitLab publish very detailed post-mortems?

I don't know but perhaps they excluded damages in their ToS?

Re: Update about the October 4th outage

#112

Post by cik »

laegooose wrote: post #105
badtux wrote: post #91

Earlier quoted context omitted.

Having been on the team that issued postmortems before, I can tell you that we said as little as possible in as vague a way as possible while meeting our minimum legal requirements. Actual Facebook customers (i.e. those who pay money to Facebook) will get a slightly more detailed release. But the whole goal is to give as little information as possible while appearing to be open. As an engineer that makes me growl, bu…

How would you explain that AWS, GCE, Cloudflare, GitLab publish very detailed post-mortems?

It's a marketing strategy. Their target customer segment is technical. FaceBooks and Twitter's for the most part, aren't.

Re: Update about the October 4th outage

#113

Post by retSava »

bawolff wrote: post #99
geerlingguy wrote: post #14

Gotta love how painfully vague this is. Sounds like a PR piece for investors, not an engineering blog piece.

It clearly is a PR piece for investors and customers. And that's ok, not everything is an eng blog.

Pointing out that this is published under _engineering_.fb.com.

Re: Update about the October 4th outage

#115

Post by Closi »

laegooose wrote: post #105
badtux wrote: post #91

Earlier quoted context omitted.

Having been on the team that issued postmortems before, I can tell you that we said as little as possible in as vague a way as possible while meeting our minimum legal requirements. Actual Facebook customers (i.e. those who pay money to Facebook) will get a slightly more detailed release. But the whole goal is to give as little information as possible while appearing to be open. As an engineer that makes me growl, bu…

How would you explain that AWS, GCE, Cloudflare, GitLab publish very detailed post-mortems?

These companies average users are highly technical developers, while facebooks users are from a much wider demographic.

It’s not really surprising to me that Facebook is writing comms that most users will understand right now, rather than publishing detailed post-mortems straight away. You have to speak the same language as your users initially in these comms.

Although I wouldn’t be surprised if we see a post-mortem in the days ahead, but Facebook probably will want to say why it happened (not just what happened, but why did it not get detected during testing, was the configuration change correct but there is an underlying bug on the routers etc) and what new mitigation’s will be put in place to stop it happening again, and these might not be known yet.

Re: Update about the October 4th outage

#116

Post by pbhjpbhj »

bawolff wrote: post #103
tomrod wrote: post #30

Earlier quoted context omitted.

Like what?

Lots of people blaming dns

I think mainstream news sources either used "DNS" as a term for routing internet traffic because most people haven't heard of BGP or come across autonomous systems (AS), or because they genuinely thought reports of routing issues meant it was DNS.

Re: Update about the October 4th outage

#117

Post by greiskul »

tshaddox wrote: post #78
spoonjim wrote: post #49

Earlier quoted context omitted.

Single point of failure is the oldest anti-pattern in the book. Goes back 5000 years.

Is there evidence to suggest that redundancy could have solved this problem? If this was simply a bad configuration that was propagated (as intended) to a large number of systems, I would hardly call that a single point of failure.

Yeah, I don't think redundancy would have prevented this. What could prevent this is stuff like canaries. Or automatic rollbacks for this kind of change, with the change having to be submitted in two steps, one where it is applied, and one where it is manually confirmed that the change worked (or automatically by a system outside the network). If no confirmation is given after a certain time, the system should revert to the old configuration.

Of course, we can't really know for sure until/if they release exactly what caused this issue.

Re: Update about the October 4th outage

#119

Post by gizmo385 »

bawolff wrote: post #101
padolsey wrote: post #81

Earlier quoted context omitted.

I mean, not an RCA per se, but info more akin to cloudflare's blog post would be v welcome IMHO: https://blog.cloudflare.com/october-2021-facebook-outage/

Both posts have essentially the same info - the fb one just didn't include an explainer on how the internet works.

The Cloudflare post includes graphs depicting how the issue looked downstream and a rough initial timeline for the incident. The Facebook post says basically nothing more than that there was a networking mistake.

I hope we get something more substantial and informative than that over the next couple of weeks, but it doesn't seem (at least from my searching) that Facebook is in the business of publicly posting in-depth post-mortems for their outages, which I personally find unfortunate.

Post reply on HN