Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

Post reply on HN
Print view View original

91–100 of 239 posts

Re: Update about the October 4th outage

#91

Post by badtux »

andrewxdiamond wrote: post #8

This more or less confirms what we’ve heard, and I appreciate the speed, but it’s incredibly lame from a details point of view. Will a real postmortem follow? Or is this the best we are gonna get?

Having been on the team that issued postmortems before, I can tell you that we said as little as possible in as vague a way as possible while meeting our minimum legal requirements. Actual Facebook customers (i.e. those who pay money to Facebook) will get a slightly more detailed release. But the whole goal is to give as little information as possible while appearing to be open. As an engineer that makes me growl, but that's how it is in this litigous world -- don't want to give someone a reason to sue.

Re: Update about the October 4th outage

#92

Post by switch007 »

Thorrez wrote: post #69
colechristensen wrote: post #44

Earlier quoted context omitted.

I entirely believed that the badge readers on doors were broken, I have set systems like that up and they’re network connected. It is not the least bit surprising that BGP chaos would break that. That doesn’t mean people literally couldn’t get in the building, more like security was really annoyed and confused all day.

A New York Times reporter said employees had trouble getting in, and I don't see a retraction. So I believe that is true: https://twitter.com/sheeraf/status/1445099150316503057

Do we hold journalists posting on Twitter to the same standards as writing in newspapers and their websites? Genuine question

Re: Update about the October 4th outage

#93

Post by [deleted] »

paxys wrote: post #41
colechristensen wrote: post #34

Earlier quoted context omitted.

I didn’t see any disinformation, just initial reports that it was DNS which were later explained to be caused by BGP.

- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…

[deleted]

Re: Update about the October 4th outage

#96

Post by phreeza »

sydthrowaway wrote: post #89

Any FB throwaway know if someone got fired for this?

I think it would be extremely unusual and counterproductive for someone in the trenches to get fired about this, as it is clearly a failure of procedure that this was even possible and so hard to recover from. Large companies I am aware of have a no-blame postmortem culture around this stuff. There may be people suffering consequences at a higher level in the SRE division, though I doubt this will happen in a timeframe of hours after the outage.

Re: Update about the October 4th outage

#97

Post by silisili »

0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

Not an antipattern. Route announcements/withdrawals are well, serious things and should likely flow through a single source. The idea of a 'microservice' based route update makes any network person uneasy.

Re: Update about the October 4th outage

#98

Post by [deleted] »

guerrilla wrote: post #72
paxys wrote: post #41

Earlier quoted context omitted.

- It was government intervention - Facebook was hacked - They did it on purpose to bury the whistleblower story - No one could access Facebook offices - They had to cut open servers with angle grinders - Disgruntled employees changed DNS records - Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining) They probably rushed out this blog post just to dispel some of these rum…

Most of these started out as speculation and jokes. Maybe we have a stupidity (i.e. critical thinking) problem rather than a misinformation problem. Only two of these (angle grinders and access to the offices) can really be considered lies or mistakes by an otherwise reputable source (the NYT) and one of those doesn't seem to have been refuted or retracted yet.

[deleted]

Re: Update about the October 4th outage

#100

Post by yalok »

sydthrowaway wrote: post #89

Any FB throwaway know if someone got fired for this?

The Bootcamp training at FB explicitly mentions that such things are not a fire-able offense - the attitude is around learning - if you managed to bring everything down, let’s learn together how you managed to do this… :)
Post reply on HN