What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?
AWS's us-east-1 region is experiencing issues
11–20 of 168 posts
Re: AWS's us-east-1 region is experiencing issues
#12This is why you are strongly urged not to rely on one region or AZ.
Seems like it would be conflict of interest to increase robustness of single AZ (so it never goes down or has its own redundancy) vs. increased revenues from multi AZ deployment.
What's the point of cloud if we have to manage robustness of their own infrastructure. I can understand if that's due to natural disasters and earthquakes, but the idea should be that a single AZ should never go down barring extraordinary circumstances. AWS should be auto-balancing, handling downtimes of a single AZ without the customer ever noticing it.
It might not be a good analogy, but if a single Cloudflare edge datacenter goes down, it will automatically route traffic through others. Transparent and painless to the customer. I understand AWS is huge, and different services have different redundancy mechanisms, but just conceptually it feels like they're in a conflict of interest to increase robustness of their data centers - "We told you to have multi-AZ deployment, not our fault".
Another way to put this is make sure as an AWS customer, to 3x multiply all costs + management of multi-AZ deployment into your total costs.
Re: AWS's us-east-1 region is experiencing issues
#13This is why you are strongly urged not to rely on one region or AZ.
Might be true for running stuff in different regions/AZs but if the provisioning region is down (e.g. deploying lambda@edge) one does not really have an alternative
Re: AWS's us-east-1 region is experiencing issues
#14What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?
Absolutely no way to prove this but maybe Q1 deadlines coming up and people trying to launch things and make changes?
Re: AWS's us-east-1 region is experiencing issues
#15No post body was provided.
Re: AWS's us-east-1 region is experiencing issues
#16Based on our telemetry, this started as NXDOMAINs for sqs.us-east-1.amazonaws.com beginning in modest volumes at 20:43 UTC and becoming a total outage at 20:48 UTC. Naturally, it was completely resolved by 20:57, 5 minutes before anything was posted in the "Personal Health Dashboard" in the AWS console.
It takes a while to find a Vice President, I guess.
Re: AWS's us-east-1 region is experiencing issues
#17I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing?
Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.
Re: AWS's us-east-1 region is experiencing issues
#18This is why you are strongly urged not to rely on one region or AZ.
Good advice, though AWS still has some services that don't work completely independently. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And I wish they didn't have global-looking endpoints (like https://sts.amazonaws.com) that aren't really global or resilient.
Re: AWS's us-east-1 region is experiencing issues
#19My Aruba Instant ON Ap's are "offline" (orange) even tho they work and I am online. My first tought is that some Cloud went nirvana state
Re: AWS's us-east-1 region is experiencing issues
#20Does AWS have a plan to improve this region?
Do they acknowledge the problem?
It's been a joke for years how bad us-east-1 is.