Hosting and processing all the photos for the California International Marathon on EC2, this doesnt make easier dealing with impatient customers any easier
AWS us-east-1 outage
661–670 of 1001 posts
Re: AWS us-east-1 outage
#662I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…
I don't think anecdotes like this are even worth sharing, honestly. There's so much context lost here, so much that can be lost in translation. No one should be drawing any conclusions from this post.
Re: AWS us-east-1 outage
#663...and imdb
And Netflix.
I'm surprised Netflix is down. They are multi-region: https://netflixtechblog.com/active-active-for-multi-regional...
Re: AWS us-east-1 outage
#664Surprised Netflix is down. I thought they were hot/hot multi-region:
https://netflixtechblog.com/active-active-for-multi-regional...
Re: AWS us-east-1 outage
#665Re: AWS us-east-1 outage
#666The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…
>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback.
Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?
Re: AWS us-east-1 outage
#667Earlier quoted context omitted.
I can’t turn on my lights… the future is weird
And that is why my lighting automation has a baseline req that it works 100% without the internet and preferably without a central controller.
HomeKit compatibility is a useful proxy for local API, since it's a hard requirement for HomeKit certification.
Re: AWS us-east-1 outage
#668Earlier quoted context omitted.
Is it really? How much does that scaling actually cost? And what's a workday anyway, surely you operate globally?
Scaling itself costs nothing, but saves money because you're not paying for unused capacity. The main application I run operates in 7 countries globally, but the US is the only one that has enough usage to require additional capacity during the workday. So out of 720 hours in a 30 day month, cloud scaling allows me to pay for additional capacity for only the (roughly) 160 hours that it's actually needed. It's a signi…
You are (conveniently or not) incorrectly assuming that the unit price of provisioned vs on-demand capacity is the same. It's not.
Re: AWS us-east-1 outage
#669Earlier quoted context omitted.
It’s hard to measure what five-9 is because you have to wait around until a 0.00001 occurs. Incentivizing post-mortems are absolutely critical in this case.
It's 0.001; the first 2 9's count. 5N = 99.999% 3N = 99.9% 1N5 = 95% 5N is <43m12s downtime per month.
I considered writing it as a percent but then decided against using it and moving the decimal instead. But good info for clarification.
Re: AWS us-east-1 outage
#670Some advice that may help: * Visit the console directly from another region's URL (e.g., https://us-east-2.console.aws.amazon.com/console/home?region... ). You can try this after you've successfully signed in but see the console failing to load as well. * If your AWS SSO app is hosted in a region other than us-east-1, you're probably fine to continue signing in with other accounts/roles. Of course, if all your stuff…
> Might also be a good idea to run AWS SSO in multiple regions if you're not already doing so. Is this possible? > AWS Organizations only supports one AWS SSO Region at a time. If you want to make AWS SSO available in a different Region, you must first delete your current AWS SSO configuration. Switching to a different Region also changes the URL for the user portal. [0] This seems to indicate you can only have one r…
Good call. I just assumed you could for some reason. I guess the fallback is to devise your own SSO implementation using STS in another region if needed.