Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Anybody else EC2 issues on us-east-1?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Ask HN: Anybody else EC2 issues on us-east-1?

#41

Post by dmourati »

encoderer wrote: post #29

Cronitor saw our customers first impacted at 4:29, largely recovered by 5:04. Another interesting thing is that AZ identifiers are randomized from customer-to-customer so when people report their failures, like here in this thread, it can sometimes seem like a problem is region-wide when in fact it's isolated to an AZ.

I don't follow your point about randomized AZs. I agree they are randomized and have observed and correlated region letters across multiple customers/accounts. How does that make it seem like a problem is region wide though?

Re: Ask HN: Anybody else EC2 issues on us-east-1?

#42

Post by encoderer »

dmourati wrote: post #41
encoderer wrote: post #29

Cronitor saw our customers first impacted at 4:29, largely recovered by 5:04. Another interesting thing is that AZ identifiers are randomized from customer-to-customer so when people report their failures, like here in this thread, it can sometimes seem like a problem is region-wide when in fact it's isolated to an AZ.

I don't follow your point about randomized AZs. I agree they are randomized and have observed and correlated region letters across multiple customers/accounts. How does that make it seem like a problem is region wide though?

When several people report failures, each in their own AZ, if you don't realize they're randomized, you can assume many/all AZs are impacted.

Re: Ask HN: Anybody else EC2 issues on us-east-1?

#43

Post by cagenut »

dmourati wrote: post #39
mkonecny wrote: post #38

Earlier quoted context omitted.

> It's really puzzling why everyone keeps hosting their projects in us-east-1. I've been using Amazon for 4 years, and this is the first time I hear their us-east-1 is more "cutting edge" and less reliable than all the other zones. This isn't even listed on their page describing these zones: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...

It's pretty common knowledge that us-east-1 is the most unreliable region. Also the oldest which helps explain why.

I agree its "common knowledge" but is it true? Or is it just a case that because 80+% (made up) of everyone hosts in us-east-1, thats the only one we hear about in HN (and other sites) threads?

Re: Ask HN: Anybody else EC2 issues on us-east-1?

#44

Post by rascoe »

mkonecny wrote: post #38
cstejerean wrote: post #21

us-east-1 is the region that has most issues. Every time you hear about an AWS outage it's typically just us-east-1, and it brings down half the internet. It's really puzzling why everyone keeps hosting their projects in us-east-1. The advantage of that region is that it tends to get new instance types and services first. And if you need to be on the bleeding edge, then you have to agree to deal with some risks. But…

> It's really puzzling why everyone keeps hosting their projects in us-east-1. I've been using Amazon for 4 years, and this is the first time I hear their us-east-1 is more "cutting edge" and less reliable than all the other zones. This isn't even listed on their page describing these zones: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...

The AWS Global Infrastructure guide to all the Regions is pretty informative. When you look at the table here[1], you realize pretty quickly that every idea that AWS has ever had gets an initial deployment in Northern Virginia (us-east-1).

It's also one of the reasons why so many people continue to use it. If you really need one of these services for your infrastructure, then your very likely going to be stuck using us-east-1. It may be quite some time before you get a 2nd region.

[1] https://aws.amazon.com/about-aws/global-infrastructure/regio...

Re: Ask HN: Anybody else EC2 issues on us-east-1?

#45

Post by tuna-piano »

coreyw wrote: post #36

This is exactly why you should design your services to run in multiple availability zones to mitigate issues like this. We run our most critical services in at least 3 availability zones and we are moving the rest of our services soon as well. While these problems are unfortunate, it is part of relying on Amazon to manage resources. Always plan for service degradation and look for ways to mitigate against issues like…

Some services, like AWS Redshift do not allow multi-AZ deployments.

Not very helpful when Redshift didn't have a single note in their status page for 30+ minutes after it went down.

Any bets on whether the status page will show up as Red for EC2/Redshift tomorrow? I'll take 100 to 1 odds for $1 that it won't be red.

Re: Ask HN: Anybody else EC2 issues on us-east-1?

#46

Post by dmourati »

cagenut wrote: post #43
dmourati wrote: post #39

Earlier quoted context omitted.

It's pretty common knowledge that us-east-1 is the most unreliable region. Also the oldest which helps explain why.

I agree its "common knowledge" but is it true? Or is it just a case that because 80+% (made up) of everyone hosts in us-east-1, thats the only one we hear about in HN (and other sites) threads?

I go by the outage reports published by AWS. I haven't done an exhaustive analysis but there is a trend to support the common knowledge. I spend most of my time in us-west-1 and us-west-2 and very few if any outages over the past 3 years.

Re: Ask HN: Anybody else EC2 issues on us-east-1?

#47

Post by dmourati »

encoderer wrote: post #42
dmourati wrote: post #41

Earlier quoted context omitted.

I don't follow your point about randomized AZs. I agree they are randomized and have observed and correlated region letters across multiple customers/accounts. How does that make it seem like a problem is region wide though?

When several people report failures, each in their own AZ, if you don't realize they're randomized, you can assume many/all AZs are impacted.

Got your point now. Thanks for replying.

Re: Ask HN: Anybody else EC2 issues on us-east-1?

#49

Post by cstejerean »

openmosix wrote: post #34
cstejerean wrote: post #21

us-east-1 is the region that has most issues. Every time you hear about an AWS outage it's typically just us-east-1, and it brings down half the internet. It's really puzzling why everyone keeps hosting their projects in us-east-1. The advantage of that region is that it tends to get new instance types and services first. And if you need to be on the bleeding edge, then you have to agree to deal with some risks. But…

us-east-1 pricing (for EC2, S3, etc.) is usually lower than other regions

Pricing between us-east-1 and us-west-2 has been largely the same in my experience. us-west-1 had a price premium, presumably due to higher costs in California.

Re: Ask HN: Anybody else EC2 issues on us-east-1?

#50

Post by cstejerean »

debaserab2 wrote: post #23
cstejerean wrote: post #21

us-east-1 is the region that has most issues. Every time you hear about an AWS outage it's typically just us-east-1, and it brings down half the internet. It's really puzzling why everyone keeps hosting their projects in us-east-1. The advantage of that region is that it tends to get new instance types and services first. And if you need to be on the bleeding edge, then you have to agree to deal with some risks. But…

I think you answered your own question. In the case of AWS, sometimes being "bleeding edge" means you're allowing amazon to scaffold infrastructure with a new service that you'd otherwise have to deal with yourself. For a small company, an AWS service being bleeding edge is still a lot more resilient than doing it yourself.

I doubt that the vast majority of customers in us-east-1 are using bleeding edge services. By the time you figure out how to integrate some new AWS offering into your infrastructure it has probably been rolled out to the other regions and the kinks worked out. Unless you need latest and greatest GPU offerings for DL, in which case maybe.
Post reply on HN