Payments in Amazon India is down - likely because of this.
AWS us-east-1 outage
451–460 of 1001 posts
Re: AWS us-east-1 outage
#452Haha my developer called me in panic telling that he crashed Amazon - was doing some load tests with Lambda
thank you for that big hearty laugh! :)
Re: AWS us-east-1 outage
#453It seems a bit long to fix! They probably paint themselves in a corner just like facebook few weeks ago. This make me think; Could it be that one day the internet will have a total global outage and it will take few days to recover?
This actually happened back in 2012 or so. Major AWS outage that took down big services all over the place, took a few days for some services to come fully back online.
https://aws.amazon.com/message/680342/
Re: AWS us-east-1 outage
#454I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.
No wonder IMDB https://www.imdb.com/> is down (returning 503). Sad that Amazon engineers don't implement what they teach their customers -- designing fault-tolerant and highly available systems.
Re: AWS us-east-1 outage
#455This is your regularly scheduled reminder: https://www.whoownsmyavailability.com/
Re: AWS us-east-1 outage
#456My payment processor Paddle also stopped working.
Re: AWS us-east-1 outage
#457Earlier quoted context omitted.
That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…
I firmly believe in the dictum "if you ship it you own it". That means you own all outages. It's not just an operator flubbing a command, or a bit of code that passed review when it shouldn't. It's all your dependencies that make your service work. You own ALL of them. People spend all this time threat modelling their stuff against malefactors, and yet so often people don't spend any time thinking about the threat mo…
if I were a black hat I would absolutely love GitHub and all the various language-specific package systems out there. giving me sooooo many ways to sneak arbitrary tailored malicious code into millions of installs around the world 24x7. sure, some of my attempts might get caught, or not but not lead to a valuable outcome for me. but that percentage that does? can make it worth it. its about scale and a massive parallelization of infiltration attempts. logic similar to the folks blasting out phishing emails or scam calls.
I love the ubiquity of thirdparty software from strangers, and the lack of bureaucratic gatekeepers. but I also hate it in ways. and not enough people know about the dangers of this second thing.
Re: AWS us-east-1 outage
#458Re: AWS us-east-1 outage
#459I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
Yep I can confirm that. The process when the outage is caused by you is called COE (correction of errors). I was oncall once for two teams because I was switching teams and I got 11 escalations in 2 hours. 10 of these were caused by an overly sensitive monitoring setting. The 11th was a real one. Guess which one I ignored. :)
Re: AWS us-east-1 outage
#460I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…
Well, the narrative is sort of what Amazon is asking for, heh?
The whole us-east-1 management console is gone, what is Amazon posting for the management console on their website?
"Service degradation"
It's not a degradation if it's outright down. Use the red status a little bit more often, this is a "disruption", not a "degradation".