Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Service Disruption Post Mortem

aws.amazon.com

Post reply on HN
Print view View original

91–100 of 108 posts

Re: AWS Service Disruption Post Mortem

#91

Post by rapind »

bretthopper wrote: post #67

I've been noticing a trend recently when reading about large scale failures of any system: it's never just one thing. AWS EBS outage, Fukushima, Chernobyl, even the great Chicago Fire (forgive me for comparing AWS to those events). Sure there's always a "root" cause, but more importantly, it's the related events that keep adding up to make the failure even worse. I can only imagine how many minor failures happen worl…

I've been an AWS (S3, EC2, SQS) user for over 3 years now and this article detailing their systems at a mid-level is kind of scaring me off of their platform. It just sounds so complicated and I'm not sure I want to rely on it for anything critical until I can really understand it myself.

Also, a couple other complex systems for your trend are; financial markets and commercial jets.

Re: AWS Service Disruption Post Mortem

#92

Post by dedward »

dholowiski wrote: post #81
thehodge wrote: post #3

An automatic 100% credit for 10 days usage, thats pretty good IMO

I agree, the credit is nice, although for me it amounts to about $6. The lost opportunity for me is huge... I have a QR code app, and missed the chance to have my QR code in a major print magazine ad. That's missed traffic, and missed exposure to the company who was going to use it - that was worth way more than $6 to me. Unfortunately as a hobby site, it's tough to justify multiple servers for redundancy

You seem to be saying it was worth a lot to you and not worth much to you at the same time - that makes no sense.

Re: AWS Service Disruption Post Mortem

#94

Post by amock »

epi0Bauqu wrote: post #37
amock wrote: post #32

Earlier quoted context omitted.

What would the purpose of that be?

They want to help people make better use of multiple availability zones. People may have reserved a bunch of instances, but would be better off distributing those more effectively across zones.

That makes sense. I wonder if something like the free realm transfers in World of Warcraft would make sense. Maybe the AZ mapping randomization keeps things balanced but encouraging multi-az deployments seems like a good idea.

Re: AWS Service Disruption Post Mortem

#96

Post by JasonMoyMN »

What few people seem to acknowledge is that the post-mortem doesn't matter. It may or may not help Amazon, because the next outage (which will occur, as it does in any data center) will likely have completely different causes than this one. The post-mortem does not help the consumer in the least. Whether the cause of the outage was the long series of complicated steps explained in the post-mortem, or whether it was due to Engineer X accidentally tripping over a cable, the effect is the same. The publication of the post-mortem is simply meaningless. It would make much more sense for them to publish details of their entire architecture and infrastructure.

Re: AWS Service Disruption Post Mortem

#97

Post by gord »

brown9-2 wrote: post #69
gord wrote: post #54

This article reads like nonsense - but this is not a criticism of AWS. The real problem is there is no good mathematical model of distributed behaviour, from which statistical guarantees can be made. I think we're at the limit of what the smartest people can achieve with hand crafted code. Most likely new math will give rise to new tools and languages, in which the next generation of reliable distributed systems will…

I'm curious - what about this post-mortem "reads like nonsense"?

All of it, the general approach is wrong - so the nonsense is the part where you believe your current set of abstractions about distributed networks are adequate.

Also the part where you reapply those same abstractions to fix the hole, without realizing that the problem is you simply don't yet have tools that are capable of writing a robust system - despite the evidence to the contrary.

If a day long outage of this scale is not enough to make us rethink distributed systems, what is?

Re: AWS Service Disruption Post Mortem

#99

Post by ra »

space-monkey wrote: post #73
ra wrote: post #65

Earlier quoted context omitted.

In which case they shouldn't replicate across availability zones. 'shared nothing' is the only way to islandize failures.

And that's why they have multiple fully-isolated regions. Availability zones are a purposeful tradeoff that provides easier to use service with higher inter-zone communication performance and lower cost.

No, that's not what Amazon says.

The following is from the AWS web site [1]:

> Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location.

No mention of tradeoffs.

[1] http://aws.amazon.com/ec2/

Re: AWS Service Disruption Post Mortem

#100

Post by bd_at_rivenhill »

bretthopper wrote: post #67

I've been noticing a trend recently when reading about large scale failures of any system: it's never just one thing. AWS EBS outage, Fukushima, Chernobyl, even the great Chicago Fire (forgive me for comparing AWS to those events). Sure there's always a "root" cause, but more importantly, it's the related events that keep adding up to make the failure even worse. I can only imagine how many minor failures happen worl…

Here's another good resource for understanding these types of problems:

http://www.amazon.com/Normal-Accidents-Living-High-Risk-Tech...

Post reply on HN