Facebook-owned sites were down • Hacker News Board

Re: Facebook-owned sites were down

#571

Post by kelnos » Mon, Oct 04, 2021, 6:36 PM UTC

DaiPlusPlus wrote: post #522

meragrin_ wrote: post #497

Earlier quoted context omitted.

The account has been deleted as well.

What are they afraid of? While they are sharing information that's internal/proprietary to the company, it isn't anything particularly sensitive and having some transparency into the problem is good for everyone. Who'd want to work for a company that might take disciplinary action because an SRE posted a reddit comment to basically say "BGP's down lol" - If I was in charge I'd give them a modest EOY bonus for being h…

> If I was in charge I'd give them a modest EOY bonus for being helpful in their outreach to my users in the wider community.

That seems pretty unlikely at any but the smallest of companies. Most companies unify all external communications through some kind of PR department. In those cases usually employees are expressly prohibited from making any public comments about the company without approval.

Re: Facebook-owned sites were down

#572

Post by bentcorner » Mon, Oct 04, 2021, 6:36 PM UTC

elboru wrote: post #169

Is it just me or HN also feels kinda laggy?

General tip: If HN is being laggy and you're determined you want to waste some time here, open it in a private window. HN works extremely quickly if it doesn't know who you are.

Re: Facebook-owned sites were down

#573

Post by alexdumitru » Mon, Oct 04, 2021, 6:36 PM UTC

jurajmlich wrote: post #393

It seems it has caused DNS servers crash for one of biggest Czechia's internet provider - Vodafone. Can be unrelated but I doubt it ( https://twitter.com/BlazejKrajnak/status/1445063232486531099 ). Think of it - half the country doesn't have internet because of this crash, that's terrifying. (Switching DNS servers obviously works but that's not something the general population will do)

Vodafone is down in Romania too.

Re: Facebook-owned sites were down

#574

Post by swayson » Mon, Oct 04, 2021, 6:38 PM UTC

This really is a fascinating case-study of what is truly resilient systems. More often than not, they are not centralized.

Re: Facebook-owned sites were down

#575

Post by cronix » Mon, Oct 04, 2021, 6:38 PM UTC

DaiPlusPlus wrote: post #522

meragrin_ wrote: post #497

Earlier quoted context omitted.

The account has been deleted as well.

What are they afraid of? While they are sharing information that's internal/proprietary to the company, it isn't anything particularly sensitive and having some transparency into the problem is good for everyone. Who'd want to work for a company that might take disciplinary action because an SRE posted a reddit comment to basically say "BGP's down lol" - If I was in charge I'd give them a modest EOY bonus for being h…

> What are they afraid of?

Zuckerberg Loses $7 Billion in Hours as Facebook Plunges

https://finance.yahoo.com/news/zuckerberg-loses-7-billion-ho...

Stop the hemorrhaging. Too much bad press for FB lately and it all adds up.

Re: Facebook-owned sites were down

#576

Post by jose-cl » Mon, Oct 04, 2021, 6:38 PM UTC

elboru wrote: post #169

Is it just me or HN also feels kinda laggy?

yes, me too (I'm in south-america)

Re: Facebook-owned sites were down

#577

Post by throwaway123x2 » Mon, Oct 04, 2021, 6:38 PM UTC

The cynic in me wonders if this is related to the Pandora Papers leak

Re: Facebook-owned sites were down

#578

Post by quaintdev » Mon, Oct 04, 2021, 6:39 PM UTC

bentcorner wrote: post #572

elboru wrote: post #169

Is it just me or HN also feels kinda laggy?

General tip: If HN is being laggy and you're determined you want to waste some time here, open it in a private window. HN works extremely quickly if it doesn't know who you are.

This works like charm. Thank you!

Re: Facebook-owned sites were down

#579

Post by Narushia » Mon, Oct 04, 2021, 6:39 PM UTC

guidopallemans wrote: post #476

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

He just deleted all his updates. user: https://old.reddit.com/user/ramenporn some messages: * This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team). * Will try to provide any important/interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls. * Update 1440 UTC: \ As many of you…

The 1440 UTC update is also archived on the Wayback Machine: https://web.archive.org/web/20211004171424/https://old.reddi...

And archive.today: https://archive.ph/sMgCi

Re: Facebook-owned sites were down

#580

Post by kukx » Mon, Oct 04, 2021, 6:39 PM UTC

treesknees wrote: post #523

Ueland wrote: post #493

Earlier quoted context omitted.

And there his account went poof, thanks for archiving.

They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job. Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the pe…

It is a matter of preparation. You can make sure there are KVMoIPs or other OOB technologies available on site to allow direct access from a remote location. In the worst case technician has to know how to connect the OOB device or press a power button ;)