Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

691–700 of 1001 posts

Re: AWS us-east-1 outage

#691

Post by boldman »

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevented from working and tens of million dollars of profit are simply vanishing since the start of the outage this morning. And now institutions like government and banks are throwing their entire capability into the cloud with no recourse or recovery plan. It seems bad now but I wonder how much worse it might be when no one actually has access to money because all financial traffic is going through AWS and it goes down.

We are incredibly blind to just trust just 3 cloud providers with the operational success of basically everything we do.

Why hasn't the industry come up with an alternative?

Re: AWS us-east-1 outage

#692

Post by Frost1x »

guenthert wrote: post #433
ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

Uh, four minutes to identify the root cause? Damn, those guys are on fire.

Identify or to publicly acknowledge? Chances are technical teams knew about this and noticed it fairly quickly, they've been working on the issue for some time. It probably wasn't until they identified the root cause and had a handful of strategies to mitigate with confidence that they chose to publicly acknowledge the issue to save face.

I've broken things before and been aware of it, but didn't acknowledge them until I was confident I could fix them. It allows you to maintain an image of expertise to those outside who care about the broken things but aren't savvy to what or why it's broken. Meanwhile you spent hours, days, weeks addressing the issue and suddenly pull a magic solution out of your hat to look like someone impossible to replace. Sometimes you can break and fix things without anyone even knowing which is very valuable if breaking something had some real risk to you.

Re: AWS us-east-1 outage

#693

Post by bowmessage »

boldman wrote: post #691

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

Because the majority of consumers don't know better / don't care and still buy products from companies with no backup plan. Because, really, how can any of us know better until we're burned many times over?

Re: AWS us-east-1 outage

#694

Post by tw04 »

boldman wrote: post #691

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevente…

>It seems bad now but I wonder how much worse it might be when no one actually has access to money because all financial traffic is going through AWS and it goes down.

Most financial institutions are implementing their own clouds, I can't think of any major one that is reliant on public cloud to the extent transactions would stop.

>Why hasn't the industry come up with an alternative?

You mean like building datacenters and hosting your own gear?

Re: AWS us-east-1 outage

#695

Post by taf2 »

dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

I wouldn't jump to say it's short sitedness (it is shitty) but it could be a matter of being pragmatic... It's easier to maintain the code if it is loaded at run time (think thin client browser style). This way your iot device can load the lastest code and even settings from the cloud... (advantage when the cloud is available)... I think of this less of short sitedness and more a reasonable trade off (with shitty side effects)

Re: AWS us-east-1 outage

#696

Post by AnIdiotOnTheNet »

hvgk wrote: post #681
AnIdiotOnTheNet wrote: post #673

Earlier quoted context omitted.

Yep, that's a consideration of going with cloud tech: if something goes wrong you're often powerless. At least with on-prem you know who to wake up in the middle of the night and you'll get straight-forward answers about what's going on.

Depends which provider you host your crap with. I’ve had real trouble trying to get a top tier incident even acknowledged by one of the pre cloud providers. To be fair when it’s AWS when something goes snap it’s not my problem which I’m happy about (until some wise ass at AWS hires me) :)

> Depends which provider you host your crap with.

That's what I'm saying: you host it yourself in facilities owned by your company if you're not willing to have everyone twiddle their thumbs during this sort of event. Your DR environment can be co-located or hosted elsewhere.

Re: AWS us-east-1 outage

#697

Post by itisit »

s_dev wrote: post #666
dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?

> Wouldn't that present another set of complications for a very small edge case at huge cost?

One has to crunch the numbers. What does a service outage cost your business every minute/hour/day/etc in terms of lost revenue, reputational damage, violated SLAs, and other factors? For some enterprises, it's well worth the added expense and trouble of having multi-site active-active setups that span clouds and on-prem.

Re: AWS us-east-1 outage

#698

Post by ricardobayes »

unethical_ban wrote: post #300
tgtweak wrote: post #131

Earlier quoted context omitted.

If you're not multi-cloud in 2021 and are expecting 5-9's, I feel bad for you.

If you're not multi-region, I feel bad for you. If your company is shoehorning you into using multiple clouds and learning a dozen products, IAM and CICD dialects simultaneously because "being cloud dependent is bad", I feel bad for you. Doing one cloud correctly from a current DevSecOps perspective is a multi-year ask. I estimate it takes about 25 people working full time on managing and securing infrastructure per…

Someone start devops as a service please

Re: AWS us-east-1 outage

#699

Post by outime »

taf2 wrote: post #695
dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

I wouldn't jump to say it's short sitedness (it is shitty) but it could be a matter of being pragmatic... It's easier to maintain the code if it is loaded at run time (think thin client browser style). This way your iot device can load the lastest code and even settings from the cloud... (advantage when the cloud is available)... I think of this less of short sitedness and more a reasonable trade off (with shitty sid…

Then you could just keep a local copy available as a fallback in case the latest code cannot be fetched. Not doing the bare minimum and screwing the end user isn't acceptable IMHO. But I also understand that'd take some engineer hours and report virtually no benefits as these outages are rare (not sure how Roomba's reliability is in general on the other hand) so here we are.

Re: AWS us-east-1 outage

#700

Post by thayne »

dylan604 wrote: post #657

The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS…

there's a reason it is called the Internet of things, and not the "local network of things". Even if the latter is probably what most customers would prefer.
Post reply on HN