I heard from my cousin who works at an Amazon warehouse that the conveyor belts stopped working and items were messed up and getting randomly removed off the belts.
AWS us-east-1 outage
631–640 of 1001 posts
Re: AWS us-east-1 outage
#632Some sage advice I learned a while ago: "Avoid us-east-1 as much as possible".
If you need to be up all the time don't you use more than 1 region or do you need the ultra low ping for running 9-11 operator phone systems that calculate real time firing solutions to snipe ICBMs out of low orbit?
Re: AWS us-east-1 outage
#633Earlier quoted context omitted.
It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…
Well, the narrative is sort of what Amazon is asking for, heh? The whole us-east-1 management console is gone, what is Amazon posting for the management console on their website? "Service degradation" It's not a degradation if it's outright down. Use the red status a little bit more often, this is a "disruption", not a "degradation".
Because the console works just fine in us-east-2 and that the console on the status page does not display regions.
If the console works 100% in us-east-2 and not in us-east-1 why would they put the console completely down in us-east?
Re: AWS us-east-1 outage
#634Earlier quoted context omitted.
https://downdetector.com Amazing and scary to see all the unrelated services down right now.
I think it's pretty unlikely that both Google and Facebook are affected by this minor AWS outage, whatever DownDetector says. I even did a spot check on some of the smaller websites they report as "down", like canva.com, and didn't see any issues.
You might be right about Google and Facebook, but this isn't minor at all. Impact is widespread.
Re: AWS us-east-1 outage
#635imdb seems down too and returning 503. Is it related? Here is the output. Kind of funny. D'oh! Error 503 We're sorry, something went wrong. Please try again...wait...wait...yep, try reload/refresh now. But if you are seeing this again, please report it here. Please explain which page you were at and where on it that you clicked Thank you!
IMDB belongs to Amazon, so likely on AWS too.
This also confirms it: https://downdetector.com/status/imdb/
Re: AWS us-east-1 outage
#636A former colleague told me years ago that us-east-1 is basically the guinea pig where changes get tested before being rolled out to the other regions, and as a result is less stable than the others. Does anyone know if there's any truth to this?
I can't see why they'd use the most common/popular region as a guinea pig.
Problem: you've gone as far as you can go testing internally or with test groups. You know there are edge cases you'll only identify by having enough people test it.
Solution: push it to production on the zone with the most users and see what breaks.
Re: AWS us-east-1 outage
#637Some advice that may help:
* Visit the console directly from another region's URL (e.g., https://us-east-2.console.aws.amazon.com/console/home?region...). You can try this after you've successfully signed in but see the console failing to load as well.
* If your AWS SSO app is hosted in a region other than us-east-1, you're probably fine to continue signing in with other accounts/roles.
Of course, if all your stuff is in us-east-1, you're out of luck.
EDIT: Removed incorrect advice about running AWS SSO in multiple regions.
Re: AWS us-east-1 outage
#638Re: AWS us-east-1 outage
#639We run as much as we can out of us-east-2 because it has more uptime than us-east-1, and I don't think I've ever regretted that decision.
Re: AWS us-east-1 outage
#640Earlier quoted context omitted.
If you're not multi-region, I feel bad for you. If your company is shoehorning you into using multiple clouds and learning a dozen products, IAM and CICD dialects simultaneously because "being cloud dependent is bad", I feel bad for you. Doing one cloud correctly from a current DevSecOps perspective is a multi-year ask. I estimate it takes about 25 people working full time on managing and securing infrastructure per…
There are so many things that can go wrong with a single provider, regardless of how many availability zones you are leveraging, that you cannot depend on 1 cloud provider for your uptime if you require that level of up. Example: Payment/Administrative issues, rogue employee with access, deprecated service, inter-region routing issues, root certificate compromises... the list goes on and it is certainly not limited t…
All cloud tech is proprietary.
There is no such thing as trivially setting up a secure, fully automated cloud stack, much less anything like a streamlined cloud agnostic toolset.
Deprecated services are not the discussion here. We're talking tactical availability, not strategic tools etc.
Rogue employees with access? You mean at the cloud provider or at your company? Still doesn't make sense. Cloud IAM is very difficult in large organizations, and each cloud does things differently.
I worked at fortune 100 finance on cloud security. Some things were quite dysfunctional, but the struggles and technical challenges are real and complex at a large organization. Perhaps you're working on a 50 employee greenfield startup. I'll hesitate to call you a clown as you did me, because that would be rude and dismissive of your experience (if any) in the field.