This is why you are strongly urged not to rely on one region or AZ.
Multi AZ is great and should be by default, but multi Region is expensive.
AWS's us-east-1 region is experiencing issues
31–40 of 168 posts
Re: AWS's us-east-1 region is experiencing issues
#32Based on our telemetry, this started as NXDOMAINs for sqs.us-east-1.amazonaws.com beginning in modest volumes at 20:43 UTC and becoming a total outage at 20:48 UTC. Naturally, it was completely resolved by 20:57, 5 minutes before anything was posted in the "Personal Health Dashboard" in the AWS console. It takes a while to find a Vice President, I guess.
Or perhaps triaging, root-causing, and fixing the issue is the highest-order bit?
Separate teams. We have a tiny team and even _we_ appoint a group to fix and a group or individual to do nothing but communicate.
Re: AWS's us-east-1 region is experiencing issues
#33I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.
Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running
Not familiar with 40k. Was it a similar idea to nuclear-power-as-religion from Foundation?
Re: AWS's us-east-1 region is experiencing issues
#34Somehow AWS managed to make their new status page more opaque than the old one. It's like they want you to scroll through their gigantic list so they can fix the issue before you find the right line.
Re: AWS's us-east-1 region is experiencing issues
#35Earlier quoted context omitted.
Seems like it would be conflict of interest to increase robustness of single AZ (so it never goes down or has its own redundancy) vs. increased revenues from multi AZ deployment. What's the point of cloud if we have to manage robustness of their own infrastructure. I can understand if that's due to natural disasters and earthquakes, but the idea should be that a single AZ should never go down barring extraordinary ci…
They would simply charge for the privilege. An EC2 'always on' or whatever option that enabled your instance to live migrate between availability zones would be a nice and expensive option.
Definitely. Then I wonder why we need the cloud :) if not for services (not EC2). Lot of mid-sized companies are re-evaluating: https://www.economist.com/business/2021/07/03/do-the-costs-o...
Re: AWS's us-east-1 region is experiencing issues
#36What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?
Russian war is another juicy possibility
Re: AWS's us-east-1 region is experiencing issues
#37I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.
Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running
If we want to normalize letting long term support people call themselves tech priests I'd very much appreciate it.
"What were your duties at your last position?" "Performing the daily ministrations and singing the praise of the machine god."
Re: AWS's us-east-1 region is experiencing issues
#38What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?
Increase in attrition across the industry.
A lot of institutional knowledge in these massive tech corporations is disappearing and we're starting to reach the tipping point.
Re: AWS's us-east-1 region is experiencing issues
#39Is us-east cursed or what?!
Re: AWS's us-east-1 region is experiencing issues
#40This is why you are strongly urged not to rely on one region or AZ.
Given the total amount of money I've lost due a single AZ being down, it was totally worth it to NOT go multi az or multi region so far. Multi AZ isn't that hard, but generally requires extra costs (one nat gw per az, etc...) But multi region in AWS is a royal pain in the ass. Many services (like SSO) do not play well with multi region setups, making things really complicated even if you IaCed your whole stack.
Those costs are the actual reason you are encouraged to go multi-AZ!
(I actually love that we have strategies and infrastructure for multi-region... it just tends to come up at scales and for applications where it is not justified.)