Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

Post reply on HN
Print view View original

181–190 of 239 posts

Re: Update about the October 4th outage

#181

Post by fragmede »

lionkor wrote: post #144

So this is pure conspiracy theory, but to me this could be a security issue. What if something deep in the core of your infrastructure is compromised? Everything at risk? Id ask my best engineer, hed suggest to shut it down, and the best way to do that is to literally pull the plug on what makes you public. Tell everyone we accidentally messed up a BGP and thats it. But yeah, likely not.

BGP is public routing information and multiple external sources are able to confirm that aspect of the story. It makes for a good conspiracy theory but the BGP withdrawal is as real as the Moon landing.

Re: Update about the October 4th outage

#182

Post by laurent92 »

lionkor wrote: post #144

So this is pure conspiracy theory, but to me this could be a security issue. What if something deep in the core of your infrastructure is compromised? Everything at risk? Id ask my best engineer, hed suggest to shut it down, and the best way to do that is to literally pull the plug on what makes you public. Tell everyone we accidentally messed up a BGP and thats it. But yeah, likely not.

If Facebook had been under actual attack, and defended by taking itself off the internet… that would be the most hands-on approach to security.

Re: Update about the October 4th outage

#183

Post by fragmede »

ignoramous wrote: post #133
cik wrote: post #112

Earlier quoted context omitted.

It's a marketing strategy. Their target customer segment is technical. FaceBooks and Twitter's for the most part, aren't.

Yeah, also a chance some eng from AWS/GCP/Azure leaks actual details if they lie or if public statements are inadequate.

Facebook just had an actual whistleblower go on 60-minutes last night.

Re: Update about the October 4th outage

#184

Post by oversighzed »

tayo42 wrote: post #71
paxys wrote: post #21

It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.

I work in a different social media company that has had some visible outages. Its always hilarious to see how wrong people are with their confident speculation. It's a good reminder that people online are often full of shit.

Lol I had the exact same lesson too. Saw people spewing falsehoods as facts and being misled by others. I’ve started to take everything I see online with a pinch of salt.

Re: Update about the October 4th outage

#185

Post by oversighzed »

sydthrowaway wrote: post #146
yalok wrote: post #100

Earlier quoted context omitted.

The Bootcamp training at FB explicitly mentions that such things are not a fire-able offense - the attitude is around learning - if you managed to bring everything down, let’s learn together how you managed to do this… :)

I don't think this is the case. Wasn't TechLead fired for SEV events?

Where did you hear that from? He doesn’t even say that in his video, he says he was fired for having side income on YouTube.

Re: Update about the October 4th outage

#186

Post by AnIdiotOnTheNet »

go_prodev wrote: post #162

I worked with a network engineer who misconfigured a router that was connecting a bank to it's DR site. The engineer had to drive across town to manually patch into the router to fix it. DR downtime was about an hour, but the bank fired him anyway. Given that Zuck lost a substantial amount of money, I wonder if the engineer faced any ramifications. Sidenote: I asked the bank infrastructure team why the DR site was in…

Harsh. Unless there is more to the story being fired for a mistake like that is ridiculous. Everyone fucks up occasionally, and on the scale of fuck ups I've certainly done worse than a 1hr DR site outage, as I'm sure pretty much anyone who's ever run infrastructure has. A consistent pattern of fucking up is grounds for termination, but not any one off instance unless there was an extreme level of negligence on display.

> Sidenote: I asked the bank infrastructure team why the DR site was in the same earthquake zone, and they thought I was crazy. They said if there's an earthquake we'll have bigger problems to deal with.

I bring this sort of thing up all the time in disaster planning. There are scales of disaster so big that business continuity is simply not going to be a priority.

Re: Update about the October 4th outage

#187

Post by throw0101a »

fragmede wrote: post #181
lionkor wrote: post #144

So this is pure conspiracy theory, but to me this could be a security issue. What if something deep in the core of your infrastructure is compromised? Everything at risk? Id ask my best engineer, hed suggest to shut it down, and the best way to do that is to literally pull the plug on what makes you public. Tell everyone we accidentally messed up a BGP and thats it. But yeah, likely not.

BGP is public routing information and multiple external sources are able to confirm that aspect of the story. It makes for a good conspiracy theory but the BGP withdrawal is as real as the Moon landing.

> It makes for a good conspiracy theory but the BGP withdrawal is as real as the Moon landing.

I wasn't aware that Stanley Kubrick was now in NetOps. /s

Re: Update about the October 4th outage

#188

Post by dugo »

Around the turn of the century, in a network the size of Europe, we had OOB comms to the core routers via ISDN/POTS. We experimented with mobile phones in the racks as well, much to the chagrin of the old telco guys running the PoPs.

Re: Update about the October 4th outage

#190

Post by jameshart »

laegooose wrote: post #105
badtux wrote: post #91

Earlier quoted context omitted.

Having been on the team that issued postmortems before, I can tell you that we said as little as possible in as vague a way as possible while meeting our minimum legal requirements. Actual Facebook customers (i.e. those who pay money to Facebook) will get a slightly more detailed release. But the whole goal is to give as little information as possible while appearing to be open. As an engineer that makes me growl, bu…

How would you explain that AWS, GCE, Cloudflare, GitLab publish very detailed post-mortems?

Are you under the impression that Facebook is a SaaS provider?

Facebook sells ad space, retail. The impact on their customers of the outage is ‘sorry, you couldn’t buy ads for a few hours.’

Demanding a public RCA for this is like demanding an RCA from Costco because they’re out of stock of tinned beans.

Post reply on HN