Earlier quoted context omitted.
I'm 95% sure it'll turn out to be a mundane config error somewhere
It will be DNS related.
Handy troubleshooting tool for outages: https://isitdns.com/
Meta outage
481–490 of 902 posts
Re: Meta outage
#482Earlier quoted context omitted.
You are absolutely correct. That would be a much better experience. That said, getting there strikes me as pretty challenging. Automatically detecting a down state is difficult and any detection is inevitably both error-prone and only works for things people have thought of to check for. The more complex the systems in question, the greater the odds of things going haywire. At Meta's scale, that is likely to be nearl…
But there's something off here. I wouldn't expecting to be shown as logged out when the services are down. I'd expect calls to fail with something aka 500 and an error showing "something happen edited on our side". Not all the apps going haywire.
Come on use a little imagination. DNS lookup for the db holding the shard with the user credentials disappears. Code isn’t expecting this, throws a generic 4xx because security instead of a generic 5xx (plenty of people writing auth code will take the stance all failures are presented the same as a bad password or non-existing username); caller interprets this a login failure.
Same auth system system used to validate logins to the bastions that have access to DNS. Voilá.
Re: Meta outage
#483Shortly before I received password recovery codes to my email, from all facebook accounts I have. My email requires 2FA, as does my facebook account, but when I went there and clicked "forgot password" there was an unknown email address added to my account. That shouldn't be possible.
Your mouse click broke it.
Re: Meta outage
#484Earlier quoted context omitted.
What a poor bunch of overworked human beings, with almost no control over the product they work on. Frantically following the whims of managers, reduced to labour units in this late stage capitalist hellscape.
But well paid at least. Working at Meta seems pretty shitty except for the pay from the stories I have heard.
It probably depends what team you're on, but I would not describe it as "pretty shitty." Being oncall for a 24/7 service sucks, yeah, but for my team it is one week a quarter and I haven't had any outside-of-biz-hour alarms the last few shifts. Other than that -- my work is challenging and interesting, my colleagues are friendly and smart, and my manager is decent. Not a lot to complain about.
Re: Meta outage
#485I am surprised by how many people still use Facebook around here, as my social circle deleted or stopped using Facebook a long time ago.
Re: Meta outage
#486Internet Outages Map - https://www.thousandeyes.com/outages/
Re: Meta outage
#487Anyone have a good guesstimate of how much money is "lost" when Youtube/Facebook/Instagram/Whatsapp (more?) are down each minute?
This is rough napkin math, no need to downvote if anyone knows the real number and this is way off :) Meta 2023 ad revenue was $131 billion. To make it easy, let's assume an even spread for # of users and ad revenue generation per hour/minute of the day and day of the year (which I'm sure is not the case). This would be: $358 million per day $15 million per hour $249k per minute This also assume a minute down won't b…
I didn't check your math, but your last number is probably per minute, not per hour.
Re: Meta outage
#488Earlier quoted context omitted.
Yup. That it’s related to the elections is also predictable, due to stress. Made worse in big corp due to affirmative action + lack of enough qualified candidates meeting diversity criteria. Which is inevitable when you have coarse criteria applied to such a large industry this way so quickly, as it takes decades for anyone to be qualified for the senior roles, and many years for junior/mid level, even if there were…
Your spitball is flying in a completely different universe than my spitball. Imo anytime you have layoffs that cut so deep, you cut through informal capability, knowledge and relationships that take a long time to form. If anything DEI helps create internal resilience because the personal networks end up a little different, giving you wider and more angles of coverage. HN talks about people in open source holding up…
Of course, and not breaking anything with layoffs is already hard when your sole criteria as a manager is ‘are they effective’. Which it’s never been that simple, especially in big corp, but it’s waaay more difficult now.
And so what happens when you’re required by the gov’t and leadership to also comply with coarse grained population statistics AND you can’t find qualified people that meet those statistics enough? On top of having to make layoffs?
My ex was a reasonably qualified software engineer, and even 4 years ago was getting no-interview offers because she was a woman - as explicitly stated by the recruiters.
It’s only gotten worse since then for hiring managers. She was offended because they literally didn’t seem to care if she was qualified or not.
I can provide links to signed and in force legal agreements between the DOL and Google for instance which formalize the need for this, and can point towards public records of evidence submitted to court of emails (internal) between recruiters which state the same too, btw.
Actual job qualifications (as in skills) did not enter the conversation at all. Just course grained DEI attributes.
So then they end up disproportionately cutting from the non-protected tranches (the groups that DO have to be qualified to stay) first because your stats still have to look good. I’m not saying DEI folks overall have no one qualified or hard working in them - rather, that there are little to no structural incentives for them to be. In many cases, they’re also unfireable/unlayoffable.
And eventually, the non-protected folks leave, burn out, or give up because f-this. Why do so much extra work when you literally can’t even get paid more for it, or be recognized because it will piss everyone else off?
And even if you’re superhuman on that front - everyone burns out eventually. Which is also why you tend to see what you see in Open Source.
It’s never any one decision, but stochastic movement in this direction has been relentless and inevitable.
Re: Meta outage
#489Hope it stays broken. Imagine? People would blink, confused, and slowly start to wake from a decade-long stupor.
how many people, realistically, use it daily? I think not a lot
Re: Meta outage
#490Looking at the Downdetector home page [1], it looks like many more services are having outages, not just the ones owned by Meta, including: - Google - YouTube - Google Play - T-Mobile - X (Twitter) - Discord - TikTok - Pokemon Go - Snapchat It looks like they all have the same failure point. [1] https://downdetector.com
Google was acting up for me as well, so that could be