Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

Post reply on HN
Print view View original

151–160 of 239 posts

Re: Update about the October 4th outage

#151

Post by colordrops »

lionkor wrote: post #144

So this is pure conspiracy theory, but to me this could be a security issue. What if something deep in the core of your infrastructure is compromised? Everything at risk? Id ask my best engineer, hed suggest to shut it down, and the best way to do that is to literally pull the plug on what makes you public. Tell everyone we accidentally messed up a BGP and thats it. But yeah, likely not.

Speaking of conspiracies, one that is floating around is that this was done to cover up spread of information around the Pandora Leak.

Re: Update about the October 4th outage

#152

Post by herald67 »

stormdennis wrote: post #149

The mobile whatsapp app should notify that the whatsapp servers are down and not allow you to just send messages that won't arrive for six hours

yup, they should. I couldn't send the messages as well and thought my mobile had some issues and tried rebooting it

Re: Update about the October 4th outage

#155

Post by BongoMcCat »

dijit wrote: post #85
tayo42 wrote: post #71

Earlier quoted context omitted.

I work in a different social media company that has had some visible outages. Its always hilarious to see how wrong people are with their confident speculation. It's a good reminder that people online are often full of shit.

I work in video games. It amazing how wrong people can be and how confident they are about being right. Even sometimes fighting _me_ about things _I_ designed and built. It’s quite sobering; taught me not to believe all the speculation I read.

Just out of curiosity, how often do people claim that your random number generator is broken? And then when you ask why they think that, it's an anecdote about some time when they had incredibly bad luck?

Re: Update about the October 4th outage

#157

Post by BoxOfRain »

runawaybottle wrote: post #33
tomrod wrote: post #30

Earlier quoted context omitted.

Like what?

We almost went down the ‘this is a subterfuge to delete whistleblower evidence’ rabbit hole.

I could have believed it given Facebook’s record for unpleasantness, but I think the fact it must have cost them tens of millions in advertising revenue is evidence against this. Even the closest thing we have to a cyberpunk dystopia isn’t going to throw that much money down the crapper to bury bad press.

Re: Update about the October 4th outage

#158

Post by jve »

KronisLV wrote: post #109
paxys wrote: post #37

Earlier quoted context omitted.

RCAs take time. It's best to issue vague statements during and right after an incident rather than make guesses.

> It's best to issue vague statements during and right after an incident rather than make guesses. Why? Why couldn't you just post that the RCA is still ongoing and that proper updates will follow? Otherwise all you get is meaningless fluff.

Well, because you can fail on initial assumptions. And may spread officially-false information.

This is not meaningless fluff. It may not provide info to technical persons, but valid info to other persons, as others noted (was hacked: yes/no).

Getting down to root cause takes time. It is usually multiple-things-at-once that caused X to happen. And then they must also make a decision on how to prevent X to happen again. All that must be written into RCA. It takes days not hours.

An example from my life: Service has intermittent disruptions. Antivirus activity correlated 100% with disruptions. Upon further investigation turns out that AV was just doing its job when there was less load. (And before anyone points out why on earth there is AV on such service, well, because it deals with user uploaded files)

So what should I have called out - AV is the guilty one? And then say: oh, no, false info.

Re: Update about the October 4th outage

#159

Post by Dylan16807 »

manquer wrote: post #18
0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

BGP has to converge to a single routing table. You are effectively asking is why is there a single routing table for the internet. To put in simple terms having a single routing table is what it makes it the internet we can share, otherwise it would just be a bunch of independent networks.

> You are effectively asking is why is there a single routing table for the internet.

No, they're asking why a single set of routers is in charge of announcing BGP routes for all of facebook. If you have multiple ASes, with independent configuration sources and independent routers broadcasting them, it's a lot harder to break everything at once.

Re: Update about the October 4th outage

#160

Post by Telluur »

dave333 wrote: post #62

I thought DARPA designed the internet to survive nuclear war - no single point of failure - clearly Facebook's network breaks that rule. They need a DNS of last resort that doesn't update fast.

On your single point of failure. That might be true, but certainly isn't these days.

Networks have grown so large and complex that the only reasonable way of managing them is through SDN, and a small mistake in configuration might results in a cascading effect on the whole infrastructure.

That's also true for the entire (western) internet. We've ended up with a centralized market where a few key players, e.g. cloud providers/CDNs/DNS (Amazon/Google/Microsoft/Akamai/Fastly/Cloudflare) can easily break large parts of the internet. See Akamai outage in July.

Post reply on HN