Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook-owned sites were down

facebook.com

Post reply on HN
Print view View original

371–380 of 1001 posts

Re: Facebook-owned sites were down

#371

Post by cotillion »

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

So, does anyone know where to one can buy an LTE gateway with a serial port interface? Asking for a friend.

Re: Facebook-owned sites were down

#372

Post by mopierotti »

rvnx wrote: post #349
pmlnr wrote: post #314

Earlier quoted context omitted.

> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.

I like how FB decided to send "ramenporn" as their spokesperson.

I'm worried about that person. I doubt Facebook will look kindly on breaking incident news being shared on reddit.

Re: Facebook-owned sites were down

#374

Post by i_like_apis »

derwiki wrote: post #301
i_like_apis wrote: post #159

Earlier quoted context omitted.

Great dip to buy. (I'm not a facebook zealot, but you know it will recover today or tomorrow once the DNS is sorted in an hour or two) Of course there is the whistle-blower issue too...

I don’t think stock dip is related to downtime; anecdotally, I’ve never seen a company’s stock affected by downtime (unless that downtime destroys the business)

You may be right, but theres a Reuters article about the downtime, this is making the news today. I would say Facebook is different because of their scale.

Looks like there are a few problems with fb in the news today ...

Re: Facebook-owned sites were down

#375

Post by perryizgr8 »

jliptzin wrote: post #215
drcongo wrote: post #66

Let's hope it's permanent.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

With Facebook, whatsapp and Instagram down, it feels like the entire internet is down for me.

Re: Facebook-owned sites were down

#376

Post by tiborsaas »

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

"I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical access to the peering routers and do all the configuration locally."

Hmm, could be a UI/UX bug then :)

Re: Facebook-owned sites were down

#377

Post by thih9 »

mov31tmov31t wrote: post #347
snickersnee11 wrote: post #307

Earlier quoted context omitted.

Just imagine the amount of stress on this people, hope the money really worth it.

It shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur. It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.

> It shouldn't be too stressful. (...) it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck

Earlier comment mentioned that there is a bottleneck, and that people who are physically able to solve the issue are few and that they need to be informed what to do; being one of these people sounds pretty stressful to me.

"but the people with physical access is separate (...) Part of this is also due to lower staffing in data centers due to pandemic measures", source: https://news.ycombinator.com/item?id=28749244

Re: Facebook-owned sites were down

#379

Post by gcoguiec »

rocho wrote: post #194
elboru wrote: post #169

Is it just me or HN also feels kinda laggy?

I can confirm, HN, GitHub and Slack are very slow for me as well. Google is very fast, on the other hand.

Dropping that many BGP routes will have its high latency toll on the whole internet backbone for minutes/hours, I'm not surprised. I wonder if the recent LE's DST Root CA X3 deprecation has something to do with the outage (some DC internal tool/API not accessible because its certificate is expired or something like that).

Re: Facebook-owned sites were down

#380

Post by kerng »

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Wondering how Facebook communicates now internally - most of their work streams likely depend on Facebooks systems which are all down.

Can engineers and security teams even access prod systems anymore? Like, would "Bastion" hosts be reachable?

Wonder if they use Signal and Slack now?

Post reply on HN