Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS outage summary

aws.amazon.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: AWS outage summary

#41

Post by barrkel »

Terretta wrote: post #40
btb wrote: post #38

Earlier quoted context omitted.

My general take on their writeup was that they have too many services that depend on EBS working. They should try to find ways to decouple critical services like loadbalancers and Amazon RDS from EBS.

Databases (RDS) do need storage and EBS is their tool for that. If something else was better, it should replace EBS across products. To your point, w/o knowing architecture, seems like ELB run state could likely be on ephemeral storage (if ELBs are EC2 instances) backed by configs on S3 unless run state is crucial across resets. If not instances, maybe use S3 directly, or ElastiCache.

Heterogeneity aids resilience because it reduces the risk that the same mode of failure occurs simultaneously. It's not clear to me that there should be one and only one mechanism for storage, because variety has a value in and of itself, apart from relative merits of different kinds of storage.

Re: AWS outage summary

#42

Post by nolok »

papercruncher wrote: post #8

I know there are lots of smart people working there but just look at the sheer amount of AWS offerings. Amazon certainly gets credit for quickly putting out new features and services but it makes me wonder if their pace has resulted in way too many moving parts with an intractable number of dependencies.

Actually, it always seems to come down to EBS failure, which is an "old" feature.

Re: AWS outage summary

#43

Post by davnola »

taligent wrote: post #13
mrkurt wrote: post #10

AWS sure does put out amazing post mortems. If only they'd make their status page more useful ...

How is the status page not useful ? It's so simple to understand. Green if everything is fine. Green if there are intermittent problems. Green if nothing works.

Haha. I'd love someone to release a greasemonkey/userscript for that status page to make degraded service more obvious.

Re: AWS outage summary

#44

Post by antirez »

cperciva wrote: post #37
seldo wrote: post #28

I dunno about you, but I could use a TL;DR for this: 1. They fucked up an internal DNS change and didn't notice 2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain 3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously 4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Ev…

I'm looking at this a bit differently. My reading of this is "a series of subtle and bizarre failures combined in a way which nobody could ever have anticipated". I think I'm a pretty good architect and coder, but I would never claim that I could design a system which couldn't fail in this sort of way -- in fact, "a background task is unable to complete, resulting in it gradually increasing its memory usage, ultimate…

That's the problem. As a designer your goal is not to claim that you can design unflawed system. Instead it is to use all your humility (and skills) to design stuff that are simple enough that are unlikely to fail because the complexity level reaches the limit of prevention and ability to analyze the failure modes.

I would like to know how much of the design in places like AWS is made more complex by the requirements of HA itself, but my guess is, a lot.

Re: AWS outage summary

#45

Post by digeridoo »

bcantrill wrote: post #15
lukev wrote: post #2

I am always astonished by how many layers these bugs actually have. It's easy to start out blaming AWS, but if anyone can realistically say they could have anticipated this type of issue at a system level, they're deluding themselves.

Full disclosure: I work for an AWS competitor. While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have ma…

To be honest, EBS is the special sauce that makes me stick with AWS. There are so many little things that become so much easier by having network storage. To me the cost of upgrading or migrating machines using local storage is just not worth the perceived reliability gain to me.

Re: AWS outage summary

#46

Post by antirez »

dchichkov wrote: post #35
antirez wrote: post #25

Earlier quoted context omitted.

Very wise comment IMHO. More complex = more complex failure modes. The only way to improve is Less complex. Well also probably local storage has a number of other advantages from the point of view of performance.

I don't think that there is an obvious way to make it less complex. But complexity is not inherently bad. Internet is very complex, yet it is robust. What feels wrong here, is that Amazon systems are too trusting to other Amazon internal systems.

Moving storage locally seems like a good move. When complexity is still too high and there are no ways to make it simpler (I don't think that's the case, btw) relax your requirements.

Re: AWS outage summary

#47

Post by Karunamon »

So how reliable is AWS in comparison to some of its competitors? I think there might be a slight bias whenever AWS has a problem because so many big name sites rely on them, and they're the proverbial 800lb gorilla of the cloud computing space.

How many of these massive outages are affecting its competitors that we never hear about?

Re: AWS outage summary

#48

Post by acdha »

seldo wrote: post #30
bigiain wrote: post #23

Earlier quoted context omitted.

FWIW, once Amazon decided that an availability zone is an unreliable unit (and they tell you this up front, and strongly suggest running multiple AZ architectures for anything where you require reliability), then any cascading failure mode in a single AZ is not something you'd expect them to spend too much time protecting against. Sure, the cascade from EBS faults to RDS and ELB meant this affected more of their sing…

While Amazon do indeed say that multi-AZ is the way to go, their last 3 major incidents (including last year's cloudpocalypse) have all been full-region incidents. IMHO, their biggest design problem is that they build their systems on top of each other (e.g. ELB is built on EBS and EIP). So when one system goes down, it takes down half a dozen others -- this is especially true of EBS, and especially dangerous because…

They've had exactly one true region-wide failure - that 17 minute routing failure earlier this year. Running a truly multi-AZ setup has avoided every other outage popularly reported as “the cloud is falling”.

Some services - e.g. Heroku - have lots of impacted customers but that's due to their architecture, not the underlying AWS.

Re: AWS outage summary

#49

Post by eclipticplane »

acdha wrote: post #48
seldo wrote: post #30

Earlier quoted context omitted.

While Amazon do indeed say that multi-AZ is the way to go, their last 3 major incidents (including last year's cloudpocalypse) have all been full-region incidents. IMHO, their biggest design problem is that they build their systems on top of each other (e.g. ELB is built on EBS and EIP). So when one system goes down, it takes down half a dozen others -- this is especially true of EBS, and especially dangerous because…

They've had exactly one true region-wide failure - that 17 minute routing failure earlier this year. Running a truly multi-AZ setup has avoided every other outage popularly reported as “the cloud is falling”. Some services - e.g. Heroku - have lots of impacted customers but that's due to their architecture, not the underlying AWS.

Mostly, but not entirely. Under the "Impact on Amazon Relational Database Service (RDS)" section they state that due to two other bugs, some Multi-zone RDS instances did NOT fail over, because their systems knew that the masters had stopped replaying to the standbys, but had continued to process some transactions (albeit more slowly, as the failures cascaded and amplified), and thus prevented the automatic promotion of a standby, as the standbys were now out of date.

Re: AWS outage summary

#50

Post by Terretta »

barrkel wrote: post #41
Terretta wrote: post #40

Earlier quoted context omitted.

Databases (RDS) do need storage and EBS is their tool for that. If something else was better, it should replace EBS across products. To your point, w/o knowing architecture, seems like ELB run state could likely be on ephemeral storage (if ELBs are EC2 instances) backed by configs on S3 unless run state is crucial across resets. If not instances, maybe use S3 directly, or ElastiCache.

Heterogeneity aids resilience because it reduces the risk that the same mode of failure occurs simultaneously. It's not clear to me that there should be one and only one mechanism for storage, because variety has a value in and of itself, apart from relative merits of different kinds of storage.

They have 3 mechanisms for storage, all can be accessed from an EC2 server: ephemeral disk, EBS, and S3.

That said, I agree with you. I commented similarly during the outage, noting that AWS may have too many interdependencies: http://news.ycombinator.com/item?id=4685571

Post reply on HN