Facebook-owned sites were down • Hacker News Board

Re: Facebook-owned sites were down

#421

Post by akudha » Mon, Oct 04, 2021, 5:32 PM UTC

jliptzin wrote: post #215

drcongo wrote: post #66

Let's hope it's permanent.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

I am surprised you have Netflix on the list. It would be annoying for 2 minutes, then you can simply go for a walk or read a book.

Re: Facebook-owned sites were down

#422

Post by awinter-py » Mon, Oct 04, 2021, 5:32 PM UTC

m0guz wrote: post #27

Ironic enough; status.fb.com also down.

need a status.status.fb.com to indicate the status of the status

and / or an S3 bucket with a json blob the apps can pull to at least tell users 'here's what's up'

Re: Facebook-owned sites were down

#423

Post by ivanmontillam » Mon, Oct 04, 2021, 5:33 PM UTC

huevosabio wrote: post #365

rvnx wrote: post #349

Earlier quoted context omitted.

I like how FB decided to send "ramenporn" as their spokesperson.

A particular facet I love of the internet era is journalists reporting serious events while having to use the completely absurd usernames... "A Facebook engineer in the response team, ramenporn..."

"Discussed in Hacker News, the user that goes by the 'huevosabio' handle, stated as a fact that..."

Re: Facebook-owned sites were down

#424

Post by armchairhacker » Mon, Oct 04, 2021, 5:33 PM UTC

drcongo wrote: post #66

Let's hope it's permanent.

> "Idk man, this seems like a tough issue. Maybe we should just give up"

> "Ok." - Zuckerburg

shuts down $100B company

Re: Facebook-owned sites were down

#425

Post by hdjjhhvvhga » Mon, Oct 04, 2021, 5:33 PM UTC

mopierotti wrote: post #372

rvnx wrote: post #349

Earlier quoted context omitted.

I like how FB decided to send "ramenporn" as their spokesperson.

I'm worried about that person. I doubt Facebook will look kindly on breaking incident news being shared on reddit.

That said, it will be interesting to read their post-mortem next year and compare it with what ramenporn wrote.

Re: Facebook-owned sites were down

#426

Post by busymom0 » Mon, Oct 04, 2021, 5:33 PM UTC

elboru wrote: post #169

Is it just me or HN also feels kinda laggy?

Yep. I am the developer of HN client HACK for iOS and Android and a bunch of users emailed me asking if my app was broken. Looks like something bigger is afoot.

Re: Facebook-owned sites were down

#427

Post by fsflover » Mon, Oct 04, 2021, 5:34 PM UTC

JCWasmx86 wrote: post #407

fsflover wrote: post #110

Earlier quoted context omitted.

Signal had their own share of downtime. How about going to a federated system instead of repeating the same mistakes? ( https://matrix.org )

But still a part would be down, if a server has an outage. How about a system, where every device that is used for chatting is a server at the same time? I wonder whether something like that already exists. Bundle it together with bigger servers to handle the load. If the bigger servers experience outages, the service can still continue, although a bit slower

Matrix P2P already exists.

Re: Facebook-owned sites were down

#428

Post by tinza123 » Mon, Oct 04, 2021, 5:34 PM UTC

bhartzer wrote: post #335

Facebook being down makes me think of all of those small businesses who never built websites. They rely on traffic and publicity from their Facebook pages only. It's so important to diversify, such as building a website.

Well assuming they do that, likely they will host websites on AWS or Azure. Then AWS is down, what are you gonna do?

Re: Facebook-owned sites were down

#429

Post by lmilcin » Mon, Oct 04, 2021, 5:34 PM UTC

pmlnr wrote: post #314

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.

I don't think so. I bet nobody is ever going to make that mistake at FB again after today.

Re: Facebook-owned sites were down

#430

Post by slaymaker1907 » Mon, Oct 04, 2021, 5:35 PM UTC

kerng wrote: post #380

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Wondering how Facebook communicates now internally - most of their work streams likely depend on Facebooks systems which are all down. Can engineers and security teams even access prod systems anymore? Like, would "Bastion" hosts be reachable? Wonder if they use Signal and Slack now?

If they planned ahead, they should have had their oncalls practice on the backup systems (like Signal/Slack/Zoom) before now.