Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

651–660 of 1001 posts

Re: AWS us-east-1 outage

#651
post #635

imdb seems down too and returning 503. Is it related? Here is the output. Kind of funny. D'oh! Error 503 We're sorry, something went wrong. Please try again...wait...wait...yep, try reload/refresh now. But if you are seeing this again, please report it here. Please explain which page you were at and where on it that you clicked Thank you!

IMDB belongs to Amazon, so likely on AWS too. This also confirms it: https://downdetector.com/status/imdb/

TIL Amazon owns IMDB

Re: AWS us-east-1 outage

#652
post #276
post #243

Earlier quoted context omitted.

And the guy who doesn't take responsibility gets promoted. Employees are not responsible for failures of management to set a good culture.

Not in healthy organizations, they don't.

You can work an entire career and maybe enjoy life in one healthy organization in that entire time even if you work in a variety of companies. It just isn't that common, though of course voicing the _ideals_ is very, very common.

Re: AWS us-east-1 outage

#653

Earlier quoted context omitted.

Why would I want to triple my capacity? Most people don't need to scale to a billion users overnight.

Many B2B-type applications have a lot of usage during the workday and minimal usage outside of it. No reason to keep all that capacity running 24/7 when you only need most of it for ~8 hours per weekday. The cloud is perfect for that use case.

idk man, idle hardware doesn't use all that much power.

https://www.thomas-krenn.com/en/wiki/Processor_P-states_and_...

Which is an implementation of:

https://web.eecs.umich.edu/~twenisch/papers/asplos09.pdf

Re: AWS us-east-1 outage

#654
post #637

Some advice that may help: * Visit the console directly from another region's URL (e.g., https://us-east-2.console.aws.amazon.com/console/home?region... ). You can try this after you've successfully signed in but see the console failing to load as well. * If your AWS SSO app is hosted in a region other than us-east-1, you're probably fine to continue signing in with other accounts/roles. Of course, if all your stuff…

I don't think you can run SSO in multiple regions on the same AWS account.

Re: AWS us-east-1 outage

#655

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of t…

> * Depending on the service one customer's experience is not the broad experience. Someone might be having a really bad day but 99.9% of the region is operating successfully, so there is no reason to update the overall status dashboard.

https://rachelbythebay.com/w/2019/07/15/giant/

Re: AWS us-east-1 outage

#657
The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS. If you can't connect to the cloud, fail, move on and load the app so that local things are allowed to work.

I'm guessing other IoT things suffer from this same short sitedness as well.

Re: AWS us-east-1 outage

#658
post #217

Earlier quoted context omitted.

That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…

I firmly believe in the dictum "if you ship it you own it". That means you own all outages. It's not just an operator flubbing a command, or a bit of code that passed review when it shouldn't. It's all your dependencies that make your service work. You own ALL of them. People spend all this time threat modelling their stuff against malefactors, and yet so often people don't spend any time thinking about the threat mo…

It's also a nightmare for software preservation. There's going to be a lot from this era that won't be usable 80 years from now because everything is so interdependent and impossible to archive. It's going to be as messy and irretrievable as the Web pre Internet Archive + Wayback are.

Re: AWS us-east-1 outage

#659

this is exact kinda over centralisation issues I was talking about. I'm one of first developers using AWS EC2, sure back when scaling is hard for small dev shops. In now day and age any one who is technically inclined, can figure out using the new technologies. Why even use AWS. Get something like Hetzner, Linodes please!

How are those better than using AWS/Azure/GCP/etc? I'd say the correct way to handle situations is to have things in multiple regions, and potentially multiple clouds if possible. Obviously, things like databases would be harder to keep in sync on multi cloud, but not impossible.

Re: AWS us-east-1 outage

#660
post #637

Some advice that may help: * Visit the console directly from another region's URL (e.g., https://us-east-2.console.aws.amazon.com/console/home?region... ). You can try this after you've successfully signed in but see the console failing to load as well. * If your AWS SSO app is hosted in a region other than us-east-1, you're probably fine to continue signing in with other accounts/roles. Of course, if all your stuff…

> Might also be a good idea to run AWS SSO in multiple regions if you're not already doing so.

Is this possible?

> AWS Organizations only supports one AWS SSO Region at a time. If you want to make AWS SSO available in a different Region, you must first delete your current AWS SSO configuration. Switching to a different Region also changes the URL for the user portal. [0]

This seems to indicate you can only have one region.

[0] https://docs.aws.amazon.com/singlesignon/latest/userguide/re...

Post reply on HN