Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS EC2/RDS Outage in us-east-1

status.aws.amazon.com

Post reply on HN
Print view View original

21–30 of 152 posts

Re: AWS EC2/RDS Outage in us-east-1

#21

Post by holykin »

rvz wrote: post #16

Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.

Multi AZ would have also sufficed. This appears localized to one zone, not the whole region.

Re: AWS EC2/RDS Outage in us-east-1

#22

Post by colinbartlett »

This seems to affect a broad swath of the internet, perhaps because the us-east-1 region is so popular? My side project StatusGator shows approximately 15% of the status pages we monitor (including our own) with a warn or down notice right now, a sizable spike over the baseline.

Re: AWS EC2/RDS Outage in us-east-1

#23

Post by holykin »

doiwin wrote: post #17
riffic wrote: post #11

>We are investigating connectivity issues affecting some instances in a single Availability Zone in the US-EAST-1 Region. Well there’s your problem, people. Use multiple AZs.

Easier said then done if that would mean synchronizing database and filesystem that is heavily written to.

Depends on what you use. RDS can span AZs and failover in events like this.

Re: AWS EC2/RDS Outage in us-east-1

#24

Post by meddlepal »

rvz wrote: post #16

Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.

Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. If you do need the kind of resiliency you usually engineer it for just a very specific component rather than the entire system.

An outage like this happens how often?

Edit: Looks like this is affecting a single AZ... so bit different situation, but I would agree if you're not capable of surviving a single AZ outage in 2019 then your engineering team should be replaced.

Re: AWS EC2/RDS Outage in us-east-1

#25

Post by ahallock »

rvz wrote: post #16

Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.

Yeah, but you have to weigh the cost of multi-region deployments and failsafes vs the cost of downtime. For smaller shops downtime may be acceptable.

Re: AWS EC2/RDS Outage in us-east-1

#29

Post by geggam »

meddlepal wrote: post #24
rvz wrote: post #16

Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.

Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. If you do need the kind of resiliency you usually engineer it for just a very specific component rather than the entire system. An outage like this happens how often? Edit: Looks like this is affecting a single AZ... so bit different situation, but I would agree if you're…

Or the PM team that wouldnt let the engineers do it right needs replaced.

Always CYA guys... you will pay for this if you dont.

Post reply on HN