Facebook-owned sites were down • Hacker News Board

Re: Facebook-owned sites were down

#521

Post by AlexandrB » Mon, Oct 04, 2021, 6:12 PM UTC

apexalpha wrote: post #268

jliptzin wrote: post #215

Earlier quoted context omitted.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

Weird, because over here WhatsApp is ingrained into the social fabric of your life. Couldn't imagine ever going back to texting/iMessage.

I still remember the era when the "in" messenger changed every 2-3 years: ICQ -> AIM -> MSN Messenger -> Google Chat, etc.

Changing messaging apps not the most convenient thing in the world, but it's not some kind of IT cataclysm. Plenty of WhatsApp competitors exist.

Re: Facebook-owned sites were down

#522

Post by DaiPlusPlus » Mon, Oct 04, 2021, 6:13 PM UTC

meragrin_ wrote: post #497

guidopallemans wrote: post #476

Earlier quoted context omitted.

He just deleted all his updates. user: https://old.reddit.com/user/ramenporn some messages: * This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team). * Will try to provide any important/interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls. * Update 1440 UTC: \ As many of you…

The account has been deleted as well.

What are they afraid of? While they are sharing information that's internal/proprietary to the company, it isn't anything particularly sensitive and having some transparency into the problem is good for everyone.

Who'd want to work for a company that might take disciplinary action because an SRE posted a reddit comment to basically say "BGP's down lol" - If I was in charge I'd give them a modest EOY bonus for being helpful in their outreach to my users in the wider community.

Re: Facebook-owned sites were down

#523

Post by treesknees » Mon, Oct 04, 2021, 6:14 PM UTC

Ueland wrote: post #493

guidopallemans wrote: post #476

Earlier quoted context omitted.

He just deleted all his updates. user: https://old.reddit.com/user/ramenporn some messages: * This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team). * Will try to provide any important/interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls. * Update 1440 UTC: \ As many of you…

And there his account went poof, thanks for archiving.

They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job.

Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the people who know the router configuration speaks to an organization that hasn't actually thought through all its failure modes. People aren't going to like that. It's not uncommon to have the datacenter techs with access and the actual software folks restricted, but that being the reason one of the most popular services in the world has been down for nearly 3 hours now will raise a lot of questions.

Edit: I also hope this doesn't damage prospects for more Work From Home. If they couldn't get anyone who knew the configuration in because they all live a plane ride away from the datacenters, I could see managers being reluctant to have a completely remote team for situations where clearly physical access was needed.

Re: Facebook-owned sites were down

#524

Post by [deleted] » Mon, Oct 04, 2021, 6:14 PM UTC

meragrin_ wrote: post #497

guidopallemans wrote: post #476

Earlier quoted context omitted.

He just deleted all his updates. user: https://old.reddit.com/user/ramenporn some messages: * This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team). * Will try to provide any important/interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls. * Update 1440 UTC: \ As many of you…

The account has been deleted as well.

[deleted]

Re: Facebook-owned sites were down

#525

Post by adamredwoods » Mon, Oct 04, 2021, 6:14 PM UTC

larntz wrote: post #312

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

This tweet seems to confirm it is a bgp issue... https://twitter.com/GossiTheDog/status/1445063880963674121?s...

Cloudflare also confirmed it:

https://twitter.com/jgrahamc/status/1445068309288951820

Also, the Domain name is for sale???

https://whois.domaintools.com/facebook.com

Re: Facebook-owned sites were down

#526

Post by nycdatasci » Mon, Oct 04, 2021, 6:14 PM UTC

Ignore.

Re: Facebook-owned sites were down

#527

Post by harias » Mon, Oct 04, 2021, 6:15 PM UTC

Ueland wrote: post #493

guidopallemans wrote: post #476

Earlier quoted context omitted.

He just deleted all his updates. user: https://old.reddit.com/user/ramenporn some messages: * This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team). * Will try to provide any important/interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls. * Update 1440 UTC: \ As many of you…

And there his account went poof, thanks for archiving.

Pushshift maintains archives of Reddit. You can use camas reddit search to view them.

Comments by u/ramenporn: https://camas.github.io/reddit-search/#{%22author%22:%22rame...

Re: Facebook-owned sites were down

#528

Post by scumcity » Mon, Oct 04, 2021, 6:15 PM UTC

hrm, bgp and dns. It's weird when decades old technology somehow fails like this. The main reason distributed systems is hard is because of the time component. Whenever you add timeouts to an algorithm, everything becomes orders of magnitude more difficult to reason about, as the number of states grows without bound. In any case, this is an epic outage and sad.

Re: Facebook-owned sites were down

#529

Post by gfosco » Mon, Oct 04, 2021, 6:15 PM UTC

kerng wrote: post #380

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Wondering how Facebook communicates now internally - most of their work streams likely depend on Facebooks systems which are all down. Can engineers and security teams even access prod systems anymore? Like, would "Bastion" hosts be reachable? Wonder if they use Signal and Slack now?

Actually, in this situation: Discord.

Re: Facebook-owned sites were down

#530

Post by winternett » Mon, Oct 04, 2021, 6:15 PM UTC

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Also, equally important to note, there was a massive expose on FaceBook yesterday that is reverberating across social media and news networks, and today, when I tried to make a post including the tag #deletefacebook, my post mysteriously could not be published and the page refreshed, mysteriously wiping my post...

This is possibly the equivalent of a corporate watergate if you ask me... Just my personal opinion as a developer though... Not presented as fact... But hrmmm.