Facebook outage is now the top story on CNN and Fox. Facebook stock down 5%. Facebook is not returning calls from Fox, or CNN.
Facebook-owned sites were down
811–820 of 1001 posts
Re: Facebook-owned sites were down
#812Earlier quoted context omitted.
Just imagine the amount of stress on this people, hope the money really worth it.
It shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur. It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.
Well, individuals will still stress, if anything, due to the feeling of bein personally responsible for inflicting damage.
I know someone who accidentally added a rule 'reject access to * for all authenticated users' in some stupid system where the ACL ruleset itself was covered by this *, and this person nearly collapsed when she realized even admins were shut out of the system. It required getting low level access to the underlying software to reverse engineer its ACLs and hack into the system. Major financial institution. Shit like leaves people with actual trauma.
As much as I hate fb, I really feel for the net ops guys trying to figure it all out, with the whole world watching (most of it with shadenfreude)
Re: Facebook-owned sites were down
#813Earlier quoted context omitted.
this is not new, this is everyday life with helping hands, on duty engineers, l2-l3 levels telling people with physical access which commands to run etc. etc. etc.
Then you have security issues like this where someone impersonates a client with helping hands and drains your exchanges hot wallet: https://www.huffpost.com/archive/ca/entry/canadian-bitcoins-...
The places I've seen this at had specific verification codes for this. One had a simple static code per person that the hands-on guys looked up in a physical binder on their desk. Very disaster proof.
The other ones had a system on the internal network in which they looked you up, called back on your company phone and asked for a passphrase the system showed them. Probably more secure but requires those systems to be working.
Re: Facebook-owned sites were down
#814When I worked there they were all about open source projects to build it themselves and control the service. Well, when your whole company is run on one DNS service this is going to bite you in the butt.
I only know of a handful of Saas apps they didn’t build internally. Sadly none of those will help them get out of this situation.
Re: Facebook-owned sites were down
#815Earlier quoted context omitted.
> Even in the biggest of organizations, they still have to wait for somebody to race down to the datacenter and plug his laptop into a router. I love this comment.
Imagine having the a huge portion of the digital world internationally riding on your shoulders...
Imagine that guy has this big npm repository locally with all those dodgy libraries with uncontrolled origin, in their /lib/node_modules with root permissions.
Wait, we all do, here.
Re: Facebook-owned sites were down
#816Earlier quoted context omitted.
The account has been deleted as well.
What are they afraid of? While they are sharing information that's internal/proprietary to the company, it isn't anything particularly sensitive and having some transparency into the problem is good for everyone. Who'd want to work for a company that might take disciplinary action because an SRE posted a reddit comment to basically say "BGP's down lol" - If I was in charge I'd give them a modest EOY bonus for being h…
As much as all of the curious techies here would love transparency into the problem, that doesn't actually do any good for Facebook (or anyone else) at the moment. Once everything is back online, making a full RCA available would do actual good for everyone. But I wouldn't hold my breath for that.
Re: Facebook-owned sites were down
#817Many local governments use FB to get info out.
Events like this show they should use multiple outlets instead of the big monopoly.
Alternatives like gab exist, but its incredibly hard to gain traction against the big monopolies.
Re: Facebook-owned sites were down
#818Does anyone have a reasonable guess on how much money they have already lost?
Re: Facebook-owned sites were down
#819Earlier quoted context omitted.
General tip: If HN is being laggy and you're determined you want to waste some time here, open it in a private window. HN works extremely quickly if it doesn't know who you are.
Wow this really works, thank you. What actually is the reason for it being much faster in a private window? Is there so much tracking going on in a normal window?
One of the first optimizations large/high traffic sites will do, is cache pages for logged out users. even if the cache is only valid for a minute, that's still a huge reduction in server traffic.
The cache is faster because its not having to talk to the database, and can be done at by the load balancing layers rather then the actual application layer.
Wikipedia does this too (although, via a layer to add back on the ip talkpage header).
Re: Facebook-owned sites were down
#820The media coverage and lots of the comments don't make sense to me. FB would not be so stupid and put all of their crucial DNS servers into a single autonomous system (which is now offline due to BGP issues). They operate literally dozens of datacenters around the world, and are surely not using a single AS for them - why not put secondary Nameservers there? Can someone make a sense of this?
Sounds like automation deployed a configuration update to most of Facebook's peering routers simultaneously. Something similar brought down Google in 2019.
If so, then it would simply be a BGP issue - no FB servers reachable, as all routes are down. But media+claims a combo of BGP/DNS. Hard to believe world-wide border routers, only responsible for networks containing DNS servers, are misconfigured. I am rely curious about that post-mortem :)