I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.
Wonder why almost all Amazon frontend looks like it was written in c++
AWS us-east-1 outage
701–710 of 1001 posts
Re: AWS us-east-1 outage
#702Earlier quoted context omitted.
I imagine there are very few businesses where the extra cost of going multi-cloud is smaller than the cost of being down during AWS outages.
Also, going multi-cloud will introduce more complexity which leads to more errors and more downtime. I'd rather sit this outage out than deal with daily risk of downtime because I'm infrastructure is too smart for its own good.
Depends on the criticality of the service. I mean you're right about adding complexity. But sometimes you can just take your really critical services and make sure it can completely withstand any one cloud provider outage.
Re: AWS us-east-1 outage
#703The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…
there's a reason it is called the Internet of things, and not the "local network of things". Even if the latter is probably what most customers would prefer.
There’s also no reason for an internet connected app to crash on load when there is no access to the internet services.
Re: AWS us-east-1 outage
#704The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…
I wouldn't jump to say it's short sitedness (it is shitty) but it could be a matter of being pragmatic... It's easier to maintain the code if it is loaded at run time (think thin client browser style). This way your iot device can load the lastest code and even settings from the cloud... (advantage when the cloud is available)... I think of this less of short sitedness and more a reasonable trade off (with shitty sid…
I don't think that's ever a reasonable tradeoff! Network access goes down all the time, and should be a fundamental assumption of any software.
Maybe I'm too old, but I can't imagine a seasoned dev, much less a tech lead, omitting planning for that failure mode
Re: AWS us-east-1 outage
#705The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…
>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?
Usually this refers to falling back to a different region in AWS. It's typical for systems to be deployed in multiple regions due to latency concerns, but it's also important for resiliency. What you call "a very small edge case" is occurring as we speak, and if you're vulnerable to it you could be losing millions of dollars.
Re: AWS us-east-1 outage
#706I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…
[deleted]
Re: AWS us-east-1 outage
#707Earlier quoted context omitted.
Should I be penalized if an upstream dependency, owned by another team, fails? Did I lack due diligence in choosing to accept the risk that the other team couldn't deliver? These are real problems in the micro-services world, especially since I own UI and there are dozens of teams pumping out services, and I'm at the mercy of all of them. The best I can do is gracefully fail when services don't function in a healthy…
Not penalised no, but questioned as to how well your graceful failure worked in the end. Remember: it may not be your fault, but it still is your problem.
A analogy for illustrating this is:
You get hit by a car and injured. The accident is the other driver's fault, but getting to the ER is your problem. The other driver may help and call an ambulance, but they might not even be able to help you if they also got hurt in the car crash.
Re: AWS us-east-1 outage
#708Earlier quoted context omitted.
if you cannot access the control plane to create or destroy resources, it is down (partial availability). The jobs that are running are basically zombies.
I'm right in the middle of an AWS-run training and we literally can't run the exercises because of this. let me repeat that: my AWS trainign that is run by AWS that I pay AWS for isn't working, because AWS is having control plane (or other) issues. This is several hours after the initial incident. We're doing training in us-west-2, but the identity service and other components run in us-east-1.
I’m running EKS in us-west-2. My pods use a role ARN and identity token file to get temporary credentials via STS. STS can’t return credentials right now. So my EKS cluster is “down” in the sense that I can’t bring up new pods. I only noticed because an auto-scaling event failed.
Re: AWS us-east-1 outage
#709If someone needs to get to the console, you can make a url like this:
Re: AWS us-east-1 outage
#710I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…
[deleted]