Which is a feature I hate, since it does that all the time even when I have a connection. Says there are 3 comments on a post, when I know there is more. Opening them doesnt show them, and no way to refresh. But going to the web page I can see them.
Facebook-owned sites were down
621–630 of 1001 posts
Re: Facebook-owned sites were down
#622Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…
Re: Facebook-owned sites were down
#623Re: Facebook-owned sites were down
#624Earlier quoted context omitted.
Reminds me of that episode in Mr Robot
The cost of the downtime would be
They are already looking at > $100M in ad loss, not counting reputation damage etc.
Re: Facebook-owned sites were down
#625Earlier quoted context omitted.
Cloudflare also confirmed it: https://twitter.com/jgrahamc/status/1445068309288951820 Also, the Domain name is for sale??? https://whois.domaintools.com/facebook.com
Weird banner at the top, seems like false advertising as it says a couple lines down: Expires on 2030-03-29
Re: Facebook-owned sites were down
#626Earlier quoted context omitted.
I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...
> I can't fathom how they didn't plan for this Maybe because they were planning for a million other possible things to go wrong, likely with higher probability than this. And busy with each day's pressing matters.
That’s like failure scenarios 101. That should be the second on the list, after “code change gone bad”.
Re: Facebook-owned sites were down
#627Earlier quoted context omitted.
Cloudflare also confirmed it: https://twitter.com/jgrahamc/status/1445068309288951820 Also, the Domain name is for sale??? https://whois.domaintools.com/facebook.com
Weird banner at the top, seems like false advertising as it says a couple lines down: Expires on 2030-03-29
Re: Facebook-owned sites were down
#628 > traceroute a.ns.facebook.com
traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets
1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms
2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms
3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms
4 12.242.105.110 (12.242.105.110) 11.446 ms 11.482 ms 11.328 ms
5 12.122.163.34 (12.122.163.34) 7.641 ms 7.668 ms 11.438 ms
6 cr83.sj2ca.ip.att.net (12.122.158.9) 4.025 ms 3.368 ms 3.394 ms
7 * * *
...
So they're hours into this outage and still haven't re-established connectivity to their own DNS servers.Re: Facebook-owned sites were down
#629Earlier quoted context omitted.
I can confirm, HN, GitHub and Slack are very slow for me as well. Google is very fast, on the other hand.
All running their DNS on AWS. My guess is that AWS is seeing a massive flood of failed and retried DNS requests for facebook properties, similar to what jgrahamc mentions here for Cloudflare: https://twitter.com/jgrahamc/status/1445066136547217413
The DNS entries should be cached by the browser (and the middleware), so that this problem should only happen once, but I get this constantly.
Also, I sometimes get an error message from HN, which seems to indicate that this is some backend issue which fails gracefully with a custom "We're having some trouble serving your request. Sorry!" on top of a 502 code.
It feels more like there is something else still broken.
Re: Facebook-owned sites were down
#630Earlier quoted context omitted.
And there his account went poof, thanks for archiving.
Pushshift maintains archives of Reddit. You can use camas reddit search to view them. Comments by u/ramenporn: https://camas.github.io/reddit-search/#{%22author%22:%22rame...