Live data from Hacker News

Facebook-owned sites were down

facebook.com

621–630 of 1001 posts

Re: Facebook-owned sites were down

#621
What I find weird is that there is no indication in the app that nothing is working. I just get a cached view of everything I've seen the last few days.

Which is a feature I hate, since it does that all the time even when I have a connection. Says there are 3 comments on a post, when I know there is more. Opening them doesnt show them, and no way to refresh. But going to the web page I can see them.

Re: Facebook-owned sites were down

#622

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

Uh oh that user deleted their account. Hope they are OK.

Re: Facebook-owned sites were down

#624
post #547

Earlier quoted context omitted.

Reminds me of that episode in Mr Robot

The cost of the downtime would be

Facebook 2021 revenue is around $100B. That’s $11M an hour. Since it’s peak hour for ad printing, one can assume double or triple this rate.

They are already looking at > $100M in ad loss, not counting reputation damage etc.

Re: Facebook-owned sites were down

#625

Earlier quoted context omitted.

Cloudflare also confirmed it: https://twitter.com/jgrahamc/status/1445068309288951820 Also, the Domain name is for sale??? https://whois.domaintools.com/facebook.com

Weird banner at the top, seems like false advertising as it says a couple lines down: Expires on 2030-03-29

I suspect it's an automated system triggered by DNS not resolving, and they try to "make an offer" if you follow through.

Re: Facebook-owned sites were down

#626

Earlier quoted context omitted.

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...

> I can't fathom how they didn't plan for this Maybe because they were planning for a million other possible things to go wrong, likely with higher probability than this. And busy with each day's pressing matters.

A config change gone bad?

That’s like failure scenarios 101. That should be the second on the list, after “code change gone bad”.

Re: Facebook-owned sites were down

#627

Earlier quoted context omitted.

Cloudflare also confirmed it: https://twitter.com/jgrahamc/status/1445068309288951820 Also, the Domain name is for sale??? https://whois.domaintools.com/facebook.com

Weird banner at the top, seems like false advertising as it says a couple lines down: Expires on 2030-03-29

You're right, it's misleading, thanks. Other sites (dreamhost, godaddy) don't list it as for sale.

Re: Facebook-owned sites were down

#628
There's still no connectivity to Facebook's DNS servers:

    > traceroute a.ns.facebook.com
      traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets
      1  dsldevice.attlocal.net (192.168.1.254)  0.484 ms  0.474 ms  0.422 ms
      2  107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1)  1.592 ms  1.657 ms  1.607 ms 
      3  71.148.149.196 (71.148.149.196)  1.676 ms  1.697 ms  1.705 ms
      4  12.242.105.110 (12.242.105.110)  11.446 ms  11.482 ms  11.328 ms
      5  12.122.163.34 (12.122.163.34)  7.641 ms  7.668 ms  11.438 ms
      6  cr83.sj2ca.ip.att.net (12.122.158.9)  4.025 ms  3.368 ms  3.394 ms
      7  * * *
      ...
So they're hours into this outage and still haven't re-established connectivity to their own DNS servers.

Re: Facebook-owned sites were down

#629
post #194

Earlier quoted context omitted.

I can confirm, HN, GitHub and Slack are very slow for me as well. Google is very fast, on the other hand.

All running their DNS on AWS. My guess is that AWS is seeing a massive flood of failed and retried DNS requests for facebook properties, similar to what jgrahamc mentions here for Cloudflare: https://twitter.com/jgrahamc/status/1445066136547217413

I can't see how this is the reason for HN to take 10 seconds for the response of the main page (I mean, the URL fetched from the address bar, not the subrequests the page does), as everything else downloads immediately.

The DNS entries should be cached by the browser (and the middleware), so that this problem should only happen once, but I get this constantly.

Also, I sometimes get an error message from HN, which seems to indicate that this is some backend issue which fails gracefully with a custom "We're having some trouble serving your request. Sorry!" on top of a 502 code.

It feels more like there is something else still broken.

Re: Facebook-owned sites were down

#630
post #527
post #493

Earlier quoted context omitted.

And there his account went poof, thanks for archiving.

Pushshift maintains archives of Reddit. You can use camas reddit search to view them. Comments by u/ramenporn: https://camas.github.io/reddit-search/#{%22author%22:%22rame...

PushShift is one of the most amazing resources out the for social media data and more people should know about it
Post reply on HN