Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS outage summary

aws.amazon.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: AWS outage summary

#61

Post by acdha »

eclipticplane wrote: post #49
acdha wrote: post #48

Earlier quoted context omitted.

They've had exactly one true region-wide failure - that 17 minute routing failure earlier this year. Running a truly multi-AZ setup has avoided every other outage popularly reported as “the cloud is falling”. Some services - e.g. Heroku - have lots of impacted customers but that's due to their architecture, not the underlying AWS.

Mostly, but not entirely. Under the "Impact on Amazon Relational Database Service (RDS)" section they state that due to two other bugs, some Multi-zone RDS instances did NOT fail over, because their systems knew that the masters had stopped replaying to the standbys, but had continued to process some transactions (albeit more slowly, as the failures cascaded and amplified), and thus prevented the automatic promotion…

I agree that's a problem but calling it a region-wide failure is an unhelpful exaggeration, particularly in the case of a SQL database where there's always a tradeoff between availability and consistency.

Re: AWS outage summary

#62

Post by oijaf888 »

jwr wrote: post #57

I avoid EBS because I think it is very complex, hard to do right, and has nasty failure modes if you use it within a UNIX environment (your code basically hangs, with no warning). Now I learned that ELB uses EBS internally. I consider this very bad news, as I inadvertently became dependent on EBS. I intend to stop using ELB.

How do you do databases? Just keep everything in instance storage and backup to S3 frequently and accept that there's possible dataloss in between backup times?

Re: AWS outage summary

#63

Post by amalag »

Terretta wrote: post #40
btb wrote: post #38

Earlier quoted context omitted.

My general take on their writeup was that they have too many services that depend on EBS working. They should try to find ways to decouple critical services like loadbalancers and Amazon RDS from EBS.

Databases (RDS) do need storage and EBS is their tool for that. If something else was better, it should replace EBS across products. To your point, w/o knowing architecture, seems like ELB run state could likely be on ephemeral storage (if ELBs are EC2 instances) backed by configs on S3 unless run state is crucial across resets. If not instances, maybe use S3 directly, or ElastiCache.

Would be very cool to have s3 snapshots available for ephemeral instances, but I am guessing that is not very practical.

Re: AWS outage summary

#65

Post by jwr »

oijaf888 wrote: post #62
jwr wrote: post #57

I avoid EBS because I think it is very complex, hard to do right, and has nasty failure modes if you use it within a UNIX environment (your code basically hangs, with no warning). Now I learned that ELB uses EBS internally. I consider this very bad news, as I inadvertently became dependent on EBS. I intend to stop using ELB.

How do you do databases? Just keep everything in instance storage and backup to S3 frequently and accept that there's possible dataloss in between backup times?

I don't do databases, for the most part. I am lucky enough to have an application that can do with very little transactional state, which we keep in redis (using instance storage), replicated to another instance and backed up to S3.

Re: AWS outage summary

#66

Post by IheartApplesDix »

Karunamon wrote: post #47

So how reliable is AWS in comparison to some of its competitors? I think there might be a slight bias whenever AWS has a problem because so many big name sites rely on them, and they're the proverbial 800lb gorilla of the cloud computing space. How many of these massive outages are affecting its competitors that we never hear about?

Well there's two problems with your question. One is that there is no other massive cloud service to compare with AWS. Every other large hosting solution is more classical with managed hardware and maybe virts being the most high-level offering for most of competitors. Nobody important uses MS or Google's cloud, and if they do, they don't like talking about it because they probably feel it gives them a competitive advantage and nobody notices when they go down anyway.
Post reply on HN