Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3 Having Problems Again? (Monday 12PM Pacific)

news.ycombinator.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: AWS S3 Having Problems Again? (Monday 12PM Pacific)

#51

Post by ceejayoz »

wbharding wrote: post #47

Indeed. As I write this we're in the midst of our third S3 outage of the day. The past two were eventually documented on the AWS Service Dashboard. The latest one has not yet received its tiny status icon to indicate an outage. It's one thing that S3 keeps going down today; we run our own server cluster and I accept that 100% uptime isn't possible. But it's aggravating that they can't at least figure out how to give…

It shouldn't be, but I've found https://twitter.com/ylastic by far the best way to find early info on AWS issues.

Re: AWS S3 Having Problems Again? (Monday 12PM Pacific)

#52

Post by toomuchtodo »

edgan wrote: post #31

The us-east-1 region gets treated differently than all other regions by AWS. Part of the reason it gets treated differently it is the default, and hence the most popular. It also doesn't help that it is on the east coast, and experiences more weather. For the above reasons, and that I work in the SF bay area, I put everything in us-west-2. us-west-2 sometimes has it's own issues, but nothing quite at the level of us-…

us-east-1 data is replicated to another region on the west coast behind the scenes. Its supposed to fail over automatically.

http://shlomoswidler.com/2009/12/read-after-write-consistenc...

> Aha! I had forgotten about the way Amazon defines its S3 regions. US-Standard has servers on both the east and west coasts (remember, this is S3 not EC2) in the same logical “region”. The engineering challenges in providing read-after-write consistency in a smaller geographical area are greatly magnified when that area is expanded. The fundamental physical limitation is the speed of light, which takes at least 16 milliseconds to cross the US coast-to-coast (that’s in a vacuum – it takes at least four times as long over the internet due to the latency introduced by routers and switches along the way).

Re: AWS S3 Having Problems Again? (Monday 12PM Pacific)

#53

Post by RyanGWU82 »

deanCommie wrote: post #29
Negitivefrags wrote: post #18

I'm risking being inflammatory here, but do people really believe that they get better uptime from AWS compared to renting dedicated servers? I feel like AWS has way too many moving parts to be stable. It's very tempting for them to reuse bits of infrastructure everywhere which increases the chances that if something goes wrong somewhere it will break your stuff. So for example, hosting instance images on S3 means th…

AWS is so massive that even when 0.1% of the customers are having problems, it is huge news like this. The reality is most customers are not affected, and overall service uptime is highest anywhere around. Not to mention that whenever AWS is having issues it's always in one region at a time, and frequently a single availability zone. As long as you build your application to be AZ-tolerant, you won't run into problems…

S3 is already supposed to be AZ-tolerant. In the US Standard region, they say your data is actually replicated across multiple geographic locations in Virginia and the Pacific Northwest. That's probably true—your data may be highly durable—but availability incidents can apparently span both geographic regions.

Re: AWS S3 Having Problems Again? (Monday 12PM Pacific)

#54

Post by mnutt »

deanCommie wrote: post #29
Negitivefrags wrote: post #18

I'm risking being inflammatory here, but do people really believe that they get better uptime from AWS compared to renting dedicated servers? I feel like AWS has way too many moving parts to be stable. It's very tempting for them to reuse bits of infrastructure everywhere which increases the chances that if something goes wrong somewhere it will break your stuff. So for example, hosting instance images on S3 means th…

AWS is so massive that even when 0.1% of the customers are having problems, it is huge news like this. The reality is most customers are not affected, and overall service uptime is highest anywhere around. Not to mention that whenever AWS is having issues it's always in one region at a time, and frequently a single availability zone. As long as you build your application to be AZ-tolerant, you won't run into problems…

The reality is most customers are not affected, and overall service uptime is highest anywhere around.

Unfortunately it's really impossible to say in this case, since they don't release numbers. Informally everyone I know with S3 buckets in US-Default had issues this morning.

As long as you build your application to be AZ-tolerant, you won't run into problems.

What you say about multiple AZs is true for EC2, but many other AWS services (especially EBS-backed ones) tend to go down across the entire region. If you're serious about availability, you really need to be in multiple regions.

Re: AWS S3 Having Problems Again? (Monday 12PM Pacific)

#55

Post by anko »

deanCommie wrote: post #29
Negitivefrags wrote: post #18

I'm risking being inflammatory here, but do people really believe that they get better uptime from AWS compared to renting dedicated servers? I feel like AWS has way too many moving parts to be stable. It's very tempting for them to reuse bits of infrastructure everywhere which increases the chances that if something goes wrong somewhere it will break your stuff. So for example, hosting instance images on S3 means th…

AWS is so massive that even when 0.1% of the customers are having problems, it is huge news like this. The reality is most customers are not affected, and overall service uptime is highest anywhere around. Not to mention that whenever AWS is having issues it's always in one region at a time, and frequently a single availability zone. As long as you build your application to be AZ-tolerant, you won't run into problems…

> As long as you build your application to be AZ-tolerant, you won't run into problems.

This is a total crock. On the 31st of July, our ec2 instances across 2 availability zones were shutdown without warning. I waited 3 hours not being able to do anything. So far I've only got info from first level support and it's been "escalated".

Re: AWS S3 Having Problems Again? (Monday 12PM Pacific)

#56

Post by akamaka »

cbsmith wrote: post #45
akamaka wrote: post #28

Earlier quoted context omitted.

Yes, your comment is unneccessarily inflammatory, in case you're wondering why I'm downvoting you. Nobody in this thread has made any comments about their uptime expectations before you brought up the subject. There's no sign here of the "people" you are referring to.

It's inflammatory to ask a question that is no doubt is some people's minds at times like this? It's not inflammatory to ask a question that people are no doubt thinking, even if the answer is obvious.

Colo versus cloud is not a typicial flamebait subject, but the way he worded it may as well have been trolling.

For example, think of someone writing "Do people really believe that Vi is better than Emacs?".

Yeah, I feel that starting off one's comment by bluntly stating one's incredulity at relying on the leading player in the cloud hosting industry is a bit inflammatory. There are a dozen other ways to word the question.

Re: AWS S3 Having Problems Again? (Monday 12PM Pacific)

#57

Post by mdellabitta »

edgan wrote: post #31

The us-east-1 region gets treated differently than all other regions by AWS. Part of the reason it gets treated differently it is the default, and hence the most popular. It also doesn't help that it is on the east coast, and experiences more weather. For the above reasons, and that I work in the SF bay area, I put everything in us-west-2. us-west-2 sometimes has it's own issues, but nothing quite at the level of us-…

IIRC, the AWS console itself is hosted out of us-east-1. Which means you're always somewhat exposed to whatever failure modes it has.

Re: AWS S3 Having Problems Again? (Monday 12PM Pacific)

#58

Post by not_kurt_godel »

mdellabitta wrote: post #57
edgan wrote: post #31

The us-east-1 region gets treated differently than all other regions by AWS. Part of the reason it gets treated differently it is the default, and hence the most popular. It also doesn't help that it is on the east coast, and experiences more weather. For the above reasons, and that I work in the SF bay area, I put everything in us-west-2. us-west-2 sometimes has it's own issues, but nothing quite at the level of us-…

IIRC, the AWS console itself is hosted out of us-east-1. Which means you're always somewhat exposed to whatever failure modes it has.

This is no longer true.

Re: AWS S3 Having Problems Again? (Monday 12PM Pacific)

#59

Post by dkuebric »

astroteller wrote: post #32
dkuebric wrote: post #6

Yep, same. Lots of latency too--here's what we're seeing: http://kuebri.ch/bucket/s3_latency_081015.png

>This webpage is not available Heh.

I'd wanted to put this on imgur instead of my micro instance, but I'm guessing imgur is backed by S3 because their upload was broken :)
Post reply on HN