Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS outage summary

aws.amazon.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: AWS outage summary

#51

Post by moe »

signifiers wrote: post #16
helper wrote: post #11

> We are already in the process of making a few changes to reduce the interdependency between ELB and EBS to avoid correlated failure in future events and allow ELB recovery even when there are EBS issues within an Availability Zone. This is music to my ears. We switched away from ELBs because of this dependency. Hopefully this statement means Amazon is working on completely removing any use of EBS from ELBs. We came…

I could not agree more, and have had zero downtime since the fleet-wide reboot last December from any of 20+ instance-backed VMs in US-East. Joe Stump of SimpleGeo and Sprint.ly and many others have come to the same conclusion. While we have a couple of RDS instances, nothing is production critical. And this: "the root cause of the Multi-AZ [MySql|Oracle|SqlServer] failures we observed during this event will be addre…

If you eschew the AWS EBS-backed services, which is pretty much all of them, why are you on EC2 to begin with?

When you operate at a scale where the above matters, and unless you need enormous elasticity, running Eucalyptus/SolusVM on your own gear is significantly cheaper.

Re: AWS outage summary

#52

Post by wglb »

This is some degree of complexity.

What does it suggest when they say "learned about new failure modes". Suggesting that there are new ones not yet learned.

One wonders if somewhere internally they have a dynamic model of how all this works. If not, might be a good time to build one.

Re: AWS outage summary

#53

Post by seldo »

cperciva wrote: post #37
seldo wrote: post #28

I dunno about you, but I could use a TL;DR for this: 1. They fucked up an internal DNS change and didn't notice 2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain 3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously 4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Ev…

I'm looking at this a bit differently. My reading of this is "a series of subtle and bizarre failures combined in a way which nobody could ever have anticipated". I think I'm a pretty good architect and coder, but I would never claim that I could design a system which couldn't fail in this sort of way -- in fact, "a background task is unable to complete, resulting in it gradually increasing its memory usage, ultimate…

I certainly didn't mean to imply that they should have predicted this -- my reason for scoring the number of simultaneous issues is to indicate what a shitstorm this was.

That said, there are some genuine deep-rooted design flaws at work here, as others have pointed out, primarily Amazon's use of EBS for critical services in their own cloud.

Re: AWS outage summary

#54

Post by seldo »

acdha wrote: post #48
seldo wrote: post #30

Earlier quoted context omitted.

While Amazon do indeed say that multi-AZ is the way to go, their last 3 major incidents (including last year's cloudpocalypse) have all been full-region incidents. IMHO, their biggest design problem is that they build their systems on top of each other (e.g. ELB is built on EBS and EIP). So when one system goes down, it takes down half a dozen others -- this is especially true of EBS, and especially dangerous because…

They've had exactly one true region-wide failure - that 17 minute routing failure earlier this year. Running a truly multi-AZ setup has avoided every other outage popularly reported as “the cloud is falling”. Some services - e.g. Heroku - have lots of impacted customers but that's due to their architecture, not the underlying AWS.

I'm sorry, but that's just not true. Prior to last week's incident:

June 28th 2012 - region-wide failure due to power outage and EBS dependency issues hitting ELB and RDS: http://aws.amazon.com/message/67457/

March 15th 2012 - 22-minute, region-wide networking interruption (no status report available)

April 2011 - cloudpocalypse, also a cascading EBS failure: http://aws.amazon.com/message/65648/

Re: AWS outage summary

#55

Post by darkarmani »

cperciva wrote: post #37
seldo wrote: post #28

I dunno about you, but I could use a TL;DR for this: 1. They fucked up an internal DNS change and didn't notice 2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain 3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously 4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Ev…

I'm looking at this a bit differently. My reading of this is "a series of subtle and bizarre failures combined in a way which nobody could ever have anticipated". I think I'm a pretty good architect and coder, but I would never claim that I could design a system which couldn't fail in this sort of way -- in fact, "a background task is unable to complete, resulting in it gradually increasing its memory usage, ultimate…

That sounds a lot like people using cron for complicated tasks that repeat every 5 minutes (db queries for example). Before you know it, the jobs pile on top of each other locking the DB and spiraling out of control.

Re: AWS outage summary

#56

Post by gizmo686 »

blaines wrote: post #7
ndcrandall wrote: post #5

Everytime there is a service outage it makes me feel better about using them in the future. Every outage is actually making the project more reliable since some issuess will only manifest in production. I believe they have a great team that's very knowledgable.

I don't necessarily agree with your first two sentences, but I definitely agree with the last one. I know they're smart. I do have some concerns that they're having too much downtime. If there's one small flaw in the system it seems that the whole thing begins to fail. If they fix the problem, and it impacts the larger system in some other unknown way, a different equally crippling issue could present itself in the f…

>I do have some concerns that they're having too much downtime. If there's one small flaw in the system it seems that the whole thing begins to fail.

I think this is a case of selection bias. Most of the time, when their is a small flaw, the system transparently bypasses the flaw and the service continues uninterrupted while they fix the initial problem. Because of this, the only failures that people see are the ones where the bypassing process fails, in which case the problem affects many people.

From the point of view of a single service running off of AWS, this is a much more stable system, becuase it will provide your service much more than a system without the auto bypass infastructure. From a end-user point of view, this benefit is not so clear, because while any given service is more reliable, they tend to go down at the same time.

Re: AWS outage summary

#57

Post by jwr »

I avoid EBS because I think it is very complex, hard to do right, and has nasty failure modes if you use it within a UNIX environment (your code basically hangs, with no warning).

Now I learned that ELB uses EBS internally. I consider this very bad news, as I inadvertently became dependent on EBS. I intend to stop using ELB.

Re: AWS outage summary

#58

Post by dmpk2k »

digeridoo wrote: post #45
bcantrill wrote: post #15

Earlier quoted context omitted.

Full disclosure: I work for an AWS competitor. While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have ma…

To be honest, EBS is the special sauce that makes me stick with AWS. There are so many little things that become so much easier by having network storage. To me the cost of upgrading or migrating machines using local storage is just not worth the perceived reliability gain to me.

It's not just reliability though. The quality of I/O is also better. You don't see such slow I/O, with crazy fluctuations, with local storage.

Re: AWS outage summary

#59

Post by forensic »

mbesto wrote: post #27
blaines wrote: post #7

Earlier quoted context omitted.

I don't necessarily agree with your first two sentences, but I definitely agree with the last one. I know they're smart. I do have some concerns that they're having too much downtime. If there's one small flaw in the system it seems that the whole thing begins to fail. If they fix the problem, and it impacts the larger system in some other unknown way, a different equally crippling issue could present itself in the f…

Do you think maybe this is just a focusing illusion?[1] And therefore the utility you associate with the service is not correctly attributed. [1] http://en.wikipedia.org/wiki/Anchoring

I like this idea. But which aspect of his belief do you think might be an illusion? I'm not clear precisely what you're referring to.

Re: AWS outage summary

#60

Post by acdha »

seldo wrote: post #54
acdha wrote: post #48

Earlier quoted context omitted.

They've had exactly one true region-wide failure - that 17 minute routing failure earlier this year. Running a truly multi-AZ setup has avoided every other outage popularly reported as “the cloud is falling”. Some services - e.g. Heroku - have lots of impacted customers but that's due to their architecture, not the underlying AWS.

I'm sorry, but that's just not true. Prior to last week's incident: June 28th 2012 - region-wide failure due to power outage and EBS dependency issues hitting ELB and RDS: http://aws.amazon.com/message/67457/ March 15th 2012 - 22-minute, region-wide networking interruption (no status report available) April 2011 - cloudpocalypse, also a cascading EBS failure: http://aws.amazon.com/message/65648/

The only region-wide outage in your links is the network one I mentioned - they called it 22 minutes, I measured it as 17 on my systems. June 28th was indeed not region-wide - I have systems in every AZ and lost exactly one of them.

The other ones say “one of our Availability Zones” - which was rather my point: if you follow long-time accepted redundancy practices, you had far less - if any - downtime than people who put everything in one AZ or rely heavily on EBS volumes not failing.

Post reply on HN