401–410 of 902 posts
Earlier quoted context omitted.
iirc these all use GCP which would make sense for them all to be disrupted at the same time. I wouldn't have thought Meta was GCP reliant though?
all of them are using oauth, likely auth provider issue?
Might be related to this event? > The cut lines include Asia-Africa-Europe 1, the Europe India Gateway, Seacom and TGN-Gulf, Hong Kong-based HGC Global Communications said. It described the cuts as affecting 25% of the traffic flowing through the Red Sea. https://apnews.com/article/red-sea-undersea-cables-yemen-hou... https://www.bbc.com/news/world-middle-east-68478828.amp
I'm pretty sure that report is just mainstream media reporting this week old cut https://www.datacenterdynamics.com/en/news/at-least-one-subs... Pretty neat if a week after a cable is cut, FB falls over. Especially when most of the source of truth databases are in the US and Europe, and that sort of data flow doesn't cross the Red Sea. FB has datacenters and points of presence all over, but outside the US/EU it's alm…
https://metastatus.com/ seems to be suffering a hug of death now...
Earlier quoted context omitted.
They also do not refund spend during this time typically so wouldn't be a 1:1
They don't? I've worked in ad tech at some smaller places and we absolutely refunded spend during outages.
Frankly, I don't buy this explanation - technically or logically both from a devops,and systems architecture level - There is no way in hell a company like Meta is pushing database design changes this significant to production? We all know how many times these database architecture changes get run in staging,then even production subsets before rolling out to production at large. Should we assume the teams working to…
What are the chances this is GPT-5 that has escaped and gone rogue? Jest aside, i wonder when/if stuff like that will actually happen.
My shadow is extremely good at mimicking my every move, but I don’t live in fear that it will make the jump to the third dimension, kill me, and assume my identity. Should I?!
Earlier quoted context omitted.
A much better UX would be clear error messaging informing users that the service is down and there is no problem with their individual account. This would prevent people from panicking they've been hacked and/or unnecessarily resetting their password.
You are absolutely correct. That would be a much better experience. That said, getting there strikes me as pretty challenging. Automatically detecting a down state is difficult and any detection is inevitably both error-prone and only works for things people have thought of to check for. The more complex the systems in question, the greater the odds of things going haywire. At Meta's scale, that is likely to be nearl…
Earlier quoted context omitted.
It would have been smarter if they hosted this on a separate reliable server altogether.
It's on AWS, but likely just has issues with provisioned capacity now that it's actually being hammered. Meta has a small collection of tools on AWS to deal with large SEV0 events like these. Another one of them is a basic communication tool that does not use Meta's own servers for anything (including auth), a super basic version of the internal SEV tool.
X-Amzn-Errortype: TooManyRequestsException
X-Amzn-Requestid:
Looks like the endpoints are on aws lambda and its getting rate limited.