Facebook-owned sites were down • Hacker News Board

Re: Facebook-owned sites were down

#311

Post by drummer » Mon, Oct 04, 2021, 4:55 PM UTC

mnd999 wrote: post #78

Somebody moved fast.

And broke everything

Re: Facebook-owned sites were down

#312

Post by larntz » Mon, Oct 04, 2021, 4:55 PM UTC

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

This tweet seems to confirm it is a bgp issue...

https://twitter.com/GossiTheDog/status/1445063880963674121?s...

Re: Facebook-owned sites were down

#313

Post by euroderf » Mon, Oct 04, 2021, 4:55 PM UTC

dkarp wrote: post #171

grayhatter wrote: post #140

Earlier quoted context omitted.

0830 actually :/ But to be fair... seems like it was a good call to not do it Friday night :D

If they chose 8:30, then it must have been really risky! ;)

The Honolulu office is getting ready for a long night :)

Re: Facebook-owned sites were down

#314

Post by pmlnr » Mon, Oct 04, 2021, 4:55 PM UTC

kossTKR wrote: post #291

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

> the people with physical access is separate from the people with knowledge of [...]

Welcome to the brave new world of troubleshooting. This will seriously bite us one day.

Re: Facebook-owned sites were down

#315

Post by mcheung610 » Mon, Oct 04, 2021, 4:55 PM UTC

RedShift1 wrote: post #7

Nothing of value was lost

But positive social value was gained

Re: Facebook-owned sites were down

#316

Post by rcarmo » Mon, Oct 04, 2021, 4:55 PM UTC

wyager wrote: post #263

rcarmo wrote: post #242

Unsurprisingly, Oculus is down as well, as are most services for the VR headset. So that's 4 major properties right now.

Can you not use an Oculus headset if FB servers are down? That’s absurd.

Some preloaded apps work (like YouTube, Firefox), but stuff like settings, the lobby, etc., are very slow or display "Unable to Load" messages. Any game that relies on your friends list seems to freeze for a while, then try to carry on.

Re: Facebook-owned sites were down

#317

Post by EB66 » Mon, Oct 04, 2021, 4:55 PM UTC

Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example:

[root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2 ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached

[root@app ~]# nslookup downforeveryoneorjustme.com 1.1.1.1 Server: 1.1.1.1 Address: 1.1.1.1#53

Non-authoritative answer: Name: downforeveryoneorjustme.com Address: 172.67.166.187 Name: downforeveryoneorjustme.com Address: 104.21.91.48

[root@app ~]#

Perhaps DNS queries are skyrocketing and overwhelming some of the major public DNS servers.

Re: Facebook-owned sites were down

#318

Post by toast0 » Mon, Oct 04, 2021, 4:56 PM UTC

yawnxyz wrote: post #168

I thought Facebook, Instagram and WhatsApp ran on different infrastructure (and they've been trying for a while to align everything)? How could they all go down at the same time, if they have different teams of engineers running each product separately? Could anyone with some background (or person familiar with the matter) explain how their system's set up?

WhatsApp and Instagram are both in FB infra. As I understand it, Instagram is fairly integrated with FB services; when I left in 2019, WhatsApp was less so, it was mostly WhatsApp specific containers running with FB's container orchestration on FB machines dedicated to WhatsApp (there was and probably is some dependence on FB systems for some parts of the app, for example the server side of multimedia is mostly a FB system with some small tweaks and specific settings, but chat should be relatively isolated). Inbound connection loadbalancing is shared though.

FWIW, WhatsApp (on phones) should be resiliant to a DNS only outage, the clients contain fallback IPs to use when DNS doesn't work, and internal services don't use DNS as far as I remember.

At one time, WhatsApp had actually separate infrastructure at SoftLayer (IBM Cloud now), but that hasn't been in place for quite some time now. When I left, it was mostly just HAProxy to catch older clients with SoftLayer IPs as their DNS fallback.

Re: Facebook-owned sites were down

#319

Post by zrail » Mon, Oct 04, 2021, 4:56 PM UTC

hugops for the engineers having to deal with this. It's incredibly stressful and I personally feel like they deserve some empathy, even if I don't like Facebook.

I wonder if maybe part of the lesson will be to run the root of your authoritative DNS hierarchy on separate infrastructure with a separate domain name. Using facebook.com as your root is cool and all but when that label disappears it causes huge issues.

Re: Facebook-owned sites were down

#320

Post by Reubachi » Mon, Oct 04, 2021, 4:57 PM UTC

wyager wrote: post #263

rcarmo wrote: post #242

Unsurprisingly, Oculus is down as well, as are most services for the VR headset. So that's 4 major properties right now.

Can you not use an Oculus headset if FB servers are down? That’s absurd.

Yep.

We've come full circle, where techies are rediscovering the original hatred for the Oculus, that it is tied to a social media walled garden, for some reason.