Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS EC2/RDS Outage in us-east-1

status.aws.amazon.com

Post reply on HN
Print view View original

41–50 of 152 posts

Re: AWS EC2/RDS Outage in us-east-1

#41

Post by fenichelar »

meddlepal wrote: post #24
rvz wrote: post #16

Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.

Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. If you do need the kind of resiliency you usually engineer it for just a very specific component rather than the entire system. An outage like this happens how often? Edit: Looks like this is affecting a single AZ... so bit different situation, but I would agree if you're…

Most of the connectivity issues we were seeing were with instances in one of the us-east-1 AZs but we were seeing issues in other AZs in us-east-1 as well. Not sure why AWS is acting like this issue in only affecting one AZ.

Re: AWS EC2/RDS Outage in us-east-1

#42

Post by garchee »

rvz wrote: post #16

Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.

Nope, our EC2 and RDS instances are multi-AZ, still got affected. Might look into being multi-region though.

Re: AWS EC2/RDS Outage in us-east-1

#43

Post by mooreds »

MuffinFlavored wrote: post #39
xyst wrote: post #28

The Spinnaker project is looking more appealing with every outage. Outage detected in X provider in Y region? Deploy infrastructure to Z provider in Y region.

From their Multi-Cloud section on their homepage: > Deploy across multiple cloud providers including AWS EC2, Kubernetes Kubernetes... where?

Probably self hosted.

Re: AWS EC2/RDS Outage in us-east-1

#44

Post by bjterry »

xyst wrote: post #28

The Spinnaker project is looking more appealing with every outage. Outage detected in X provider in Y region? Deploy infrastructure to Z provider in Y region.

This outage is only affecting a single availability zone, so taking on the complexity of multiple cloud providers would not be necessary to be resilient against it. AWS best practices would already have covered you.

Re: AWS EC2/RDS Outage in us-east-1

#45

Post by herostratus101 »

xyst wrote: post #28

The Spinnaker project is looking more appealing with every outage. Outage detected in X provider in Y region? Deploy infrastructure to Z provider in Y region.

Lol, or just architect your applications to be highly available and fault-tolerant to regional failures if that's your business requirement.

Re: AWS EC2/RDS Outage in us-east-1

#46

Post by herostratus101 »

bjterry wrote: post #44
xyst wrote: post #28

The Spinnaker project is looking more appealing with every outage. Outage detected in X provider in Y region? Deploy infrastructure to Z provider in Y region.

This outage is only affecting a single availability zone, so taking on the complexity of multiple cloud providers would not be necessary to be resilient against it. AWS best practices would already have covered you.

Where does it say it's a single AZ?

Re: AWS EC2/RDS Outage in us-east-1

#47

Post by scarface74 »

rvz wrote: post #16

Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.

It’s not an issue with an entire region, it’s an issue with a single AZ in a region. If you did the bare minimum - set up your RDS with a replica in separate AZ’s, run your servers in an autoscaling group (even with just a min/max of 1) configured for separate AZ’s, used services that are multi-AZ by default - almost all of the managed servers - you could still be up.

Re: AWS EC2/RDS Outage in us-east-1

#48

Post by cstejerean »

xyst wrote: post #28

The Spinnaker project is looking more appealing with every outage. Outage detected in X provider in Y region? Deploy infrastructure to Z provider in Y region.

Or just deploy to us-west-2 (or maybe us-east-2) in the first place and then enjoy not having to do anything every time there’s an outage in us-east-1, which is where pretty much all the AWS outages are.

And if there is an issue for some reason failing over to another AWS region is a lot easier than having to fail over to another cloud provider. Outside of a very small number of cases building for multi-cloud is a lot of unnecessary work.

Re: AWS EC2/RDS Outage in us-east-1

#49

Post by judge2020 »

herostratus101 wrote: post #46
bjterry wrote: post #44

Earlier quoted context omitted.

This outage is only affecting a single availability zone, so taking on the complexity of multiple cloud providers would not be necessary to be resilient against it. AWS best practices would already have covered you.

Where does it say it's a single AZ?

Every "more" drawer/dropdown

> We are investigating connectivity issues affecting some single-AZ RDS instances in a single Availability Zone in the US-EAST-1 Region.

Re: AWS EC2/RDS Outage in us-east-1

#50

Post by scarface74 »

crb002 wrote: post #37

Curious. Lambda not effected. EC2 being physically tied to a box does introduce extra risk I hadn't thought of.

Lambda would be just as affected if you were running inside of a VPC [1] and you ignored the multiple warnings about setting up your lambda to run in only one AZ.

[1] technically your lambda never runs “inside your VPC” but it’s a colloquialism that everyone understand.

Post reply on HN