Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook-owned sites were down

facebook.com

Post reply on HN
Print view View original

361–370 of 1001 posts

Re: Facebook-owned sites were down

#361

Post by rvnx »

mov31tmov31t wrote: post #347
snickersnee11 wrote: post #307

Earlier quoted context omitted.

Just imagine the amount of stress on this people, hope the money really worth it.

It shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur. It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.

"it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts"

Well, you'd be surprised about how one person can bring everything down and/or save the day at Facebook, Cloudflare, Google, Gitlab, etc. Most people are observers/cheerleaders when there is an incident.

Re: Facebook-owned sites were down

#362

Post by gpvos »

jliptzin wrote: post #215
drcongo wrote: post #66

Let's hope it's permanent.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

Facebook is the only one of those that I regularly use. I'd like something like Google's Android to stay around. The rest I don't need.

Re: Facebook-owned sites were down

#364

Post by bluecalm »

jliptzin wrote: post #215
drcongo wrote: post #66

Let's hope it's permanent.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

Man, out of those only Netflix going down wouldn't cause a gigantic billions of dollars worth clusterfuck to people, businesses and companies. It's nice you don't use them but about everyone around does and mostly for at least some important things.

Re: Facebook-owned sites were down

#365

Post by huevosabio »

rvnx wrote: post #349
pmlnr wrote: post #314

Earlier quoted context omitted.

> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.

I like how FB decided to send "ramenporn" as their spokesperson.

A particular facet I love of the internet era is journalists reporting serious events while having to use the completely absurd usernames...

"A Facebook engineer in the response team, ramenporn..."

Re: Facebook-owned sites were down

#366

Post by rstupek »

EB66 wrote: post #317

Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example: [root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2 ;; connection timed out; trying next origin ;; connection timed out; no servers could be…

Yes same issue for me with 8.8.8.8, errors for everything but big domains.

Re: Facebook-owned sites were down

#367

Post by chasd00 »

zrail wrote: post #319

hugops for the engineers having to deal with this. It's incredibly stressful and I personally feel like they deserve some empathy, even if I don't like Facebook. I wonder if maybe part of the lesson will be to run the root of your authoritative DNS hierarchy on separate infrastructure with a separate domain name. Using facebook.com as your root is cool and all but when that label disappears it causes huge issues.

There will be so many meetings over this. If powerpoint was listed on the stock exchange i'd say now's a good time to buy hah.

Re: Facebook-owned sites were down

#368

Post by [deleted] »

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

[deleted]
Post reply on HN