Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

671–680 of 1001 posts

Re: AWS us-east-1 outage

#671

Post by bilalq »

binaryblitz wrote: post #654
bilalq wrote: post #637

Some advice that may help: * Visit the console directly from another region's URL (e.g., https://us-east-2.console.aws.amazon.com/console/home?region... ). You can try this after you've successfully signed in but see the console failing to load as well. * If your AWS SSO app is hosted in a region other than us-east-1, you're probably fine to continue signing in with other accounts/roles. Of course, if all your stuff…

I don't think you can run SSO in multiple regions on the same AWS account.

Thanks, corrected.

Re: AWS us-east-1 outage

#673

Post by AnIdiotOnTheNet »

hvgk wrote: post #587

Well I got to bugger off home early so good job Amazon. Edit: to be clear this is because I’m utterly helplessly unable to do anything at the moment.

Yep, that's a consideration of going with cloud tech: if something goes wrong you're often powerless. At least with on-prem you know who to wake up in the middle of the night and you'll get straight-forward answers about what's going on.

Re: AWS us-east-1 outage

#674

Post by Rapzid »

broknbottle wrote: post #144
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

This gets posted every time there's an AWS outage. It mind as well be a copy pasta at this point.

It's the "grandma got run over by a reindeer" of AWS outages. Really no outage thread would be complete without this anecdote.

Re: AWS us-east-1 outage

#675

Post by dang »

PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

(This was originally a reply to https://news.ycombinator.com/item?id=29473759 but I've pruned it to make the thread less top-heavy.)

Re: AWS us-east-1 outage

#677

Post by jaywalk »

kortex wrote: post #333
jaywalk wrote: post #306

Earlier quoted context omitted.

You're not wrong, but it's possible that the organization is small enough that it's just not feasible to have enough safeguards that would prevent the outages John caused. And in that case, it's probably best that John not be promoted if he can't avoid those errors.

Current co is small. We are putting in the safeguards from Day 1. Well, okay technically like day 120, the first few months were a mad dash to MVP. But now that we have some breathing room, yeah, we put a lot of emphasis on preventing outages, detecting and diagnosing outages promptly, documenting them, doing the whole 5-why's thing, and preventing them in the future. We didn't have to, we could have kept mad dashing…

Well, I started at the small company I'm currently at around day 7300, where "source control" consisted of asking the one person who was in charge of all source code for a copy of the files you needed to work on, and then giving the updated files back. He'd write down the "checked out" files on a whiteboard to ensure that two people couldn't work on the same file at the same time.

The fact that I've gotten it to the point of using git with automated build and deployment is a small miracle in itself. Not everybody gets to start from a clean slate.

Re: AWS us-east-1 outage

#678

Post by winrid »

s_dev wrote: post #666
dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?

In this case, just connect over LAN.

Re: AWS us-east-1 outage

#680

Post by [deleted] »

s_dev wrote: post #666
dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?

[deleted]
Post reply on HN