Live data from Hacker News

Facebook-owned sites were down

facebook.com

341–350 of 1001 posts

Re: Facebook-owned sites were down

#341
post #66

Let's hope it's permanent.

Of all the big tech companies Facebook is the only one where it can completely disappear overnight and my life would be completely unaffected (or possibly improved by not having to explain to people I don't use facebook, please email or text me your invitations rather than use messenger). If Google, Amazon, Netflix, Apple disappeared the story would be completely different.

Maybe Google, because of the search engine. Android: somebody will fill the void.

Messaging: people have been switching on hordes to every new free messaging system in the 90s and early 2000s, we will adapt to something else.

Netflix and video in general: same thing without the 90s/early 2000s.

Amazon: very convenient store, we'll spend a little less and somebody will fill the void.

Apple: can't say, never bought anything from them.

By the way, when I couldn't message on WA today I thought day they finally cut me off because I still didn't accept their new privacy policy from months ago :-) I resolved to wait and see for a couple of days.

Re: Facebook-owned sites were down

#342

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

> Even in the biggest of organizations, they still have to wait for somebody to race down to the datacenter and plug his laptop into a router.

I love this comment.

Re: Facebook-owned sites were down

#345
post #37

Earlier quoted context omitted.

This cannot be said enough.

On the contrary, it's said far too much. Facebook is extremely valuable for a lot of people. I dislike Facebook as much as most people on here, but saying "it's totally pointless" is silly and it's not surprising that those who say it are ignored by those who use Facebook.

lots of people are heroin dependant, the number of people hooked doesn't make it right.

At the very least you are going to need a better arguments than that following the recent data dump.

Re: Facebook-owned sites were down

#346

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Interesting that they published stuff about their BGP setup and infrastructure a few months ago - maybe a little tweak to roll backs is needed.

"... We demonstrate how this design provides us with flexible control over routing and keeps the network reliable. We also describe our in-house BGP software implementation, and its testing and deployment pipelines. These allow us to treat BGP like any other software component, enabling fast incremental updates..."

Re: Facebook-owned sites were down

#347

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Just imagine the amount of stress on this people, hope the money really worth it.

It shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur.

It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.

Re: Facebook-owned sites were down

#349
post #314

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.

I like how FB decided to send "ramenporn" as their spokesperson.

Re: Facebook-owned sites were down

#350
post #317

Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example: [root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2 ;; connection timed out; trying next origin ;; connection timed out; no servers could be…

Yeah Cloudflare have said they're being flooded with Facebook DNS retries and had to get extra hands on deck to deal with the influx.
Post reply on HN