AWS outage summary • Hacker News Board

Re: AWS outage summary

#11

Post by helper » Sat, Oct 27, 2012, 4:01 AM UTC

> We are already in the process of making a few changes to reduce the interdependency between ELB and EBS to avoid correlated failure in future events and allow ELB recovery even when there are EBS issues within an Availability Zone.

This is music to my ears. We switched away from ELBs because of this dependency. Hopefully this statement means Amazon is working on completely removing any use of EBS from ELBs.

We came to the conclusion a year and a half ago that EBS has had too many cascading failures to be trustworthy for our production systems. We now run everything on ephemeral drives and use Cassandra distributed across multiple AZs and multiple regions for data persistence.

I highly recommend getting as many servers as you can off EBS.

Re: AWS outage summary

#12

Post by sokoloff » Sat, Oct 27, 2012, 4:11 AM UTC

blaines wrote: post #7

ndcrandall wrote: post #5

Everytime there is a service outage it makes me feel better about using them in the future. Every outage is actually making the project more reliable since some issuess will only manifest in production. I believe they have a great team that's very knowledgable.

I don't necessarily agree with your first two sentences, but I definitely agree with the last one. I know they're smart. I do have some concerns that they're having too much downtime. If there's one small flaw in the system it seems that the whole thing begins to fail. If they fix the problem, and it impacts the larger system in some other unknown way, a different equally crippling issue could present itself in the f…

If there's one small flaw in the system it seems that the whole thing begins to fail.

I wonder if there is selection bias underlying that judgment? Reading all of Amazon's post-mortems of big events, it does seem that the whole thing is fragile. What I suspect is more likely true is that AWS suffers thousands of small failures every month and most are contained as designed, with no(or minuscule) customer impact. It's the ones that turn into highly visible failures that we read about.

That said, I agree with you that EBS in particular seems to have more downtime than I'd expect. (And that other services like ELB depends on it makes it cascade in a way that's hard to design highly-available systems.)

Re: AWS outage summary

#13

Post by taligent » Sat, Oct 27, 2012, 4:19 AM UTC

mrkurt wrote: post #10

AWS sure does put out amazing post mortems. If only they'd make their status page more useful ...

How is the status page not useful ? It's so simple to understand.

Green if everything is fine. Green if there are intermittent problems. Green if nothing works.

Re: AWS outage summary

#14

Post by jlgreco » Sat, Oct 27, 2012, 4:42 AM UTC

filvdg wrote: post #3

Everything is a Freaking DNS problem :)

Well, naming things is one of the 2 hard problems in computer science. ;)

Re: AWS outage summary

#15

Post by bcantrill » Sat, Oct 27, 2012, 4:47 AM UTC

lukev wrote: post #2

I am always astonished by how many layers these bugs actually have. It's easy to start out blaming AWS, but if anyone can realistically say they could have anticipated this type of issue at a system level, they're deluding themselves.

Full disclosure: I work for an AWS competitor.

While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have made this mistake with respect to network storage in the cloud; I, too, was fooled.[1]

We have learned this lesson the hard way, many times over: local storage should be local, even in a distributed system. So while we cannot predict the specifics of the next EBS failure, we can say with absolute certainty that there will be a next failure -- and that it will be one in which the magnitude of the system failure is far greater than the initial failing component or subsystem. With respect to network storage in the cloud, the only way to win is not to play.

[1] http://joyent.com/blog/network-storage-in-the-cloud-deliciou...

Re: AWS outage summary

#16

Post by signifiers » Sat, Oct 27, 2012, 5:08 AM UTC

helper wrote: post #11

> We are already in the process of making a few changes to reduce the interdependency between ELB and EBS to avoid correlated failure in future events and allow ELB recovery even when there are EBS issues within an Availability Zone. This is music to my ears. We switched away from ELBs because of this dependency. Hopefully this statement means Amazon is working on completely removing any use of EBS from ELBs. We came…

I could not agree more, and have had zero downtime since the fleet-wide reboot last December from any of 20+ instance-backed VMs in US-East. Joe Stump of SimpleGeo and Sprint.ly and many others have come to the same conclusion.

While we have a couple of RDS instances, nothing is production critical. And this: "the root cause of the Multi-AZ [MySql|Oracle|SqlServer] failures we observed during this event will be addressed" only confirms my observations from the RSS history in the dashboard, that in nearly every major EBS-related "service event" (including the ones that happen every few weeks and never get this level of post-mortem), the managed databases, load balancers and config management (beanstalk) services go down too.

When you move from AWS' basic EC2 IaaS VMs with instance (ephemeral|local) storage to EBS-backed (basically vSAN) storage, your multi-month uptime odds go down considerably. But when you step up to the PaaS of managed DBs, load balancing, dynamo, etc., yes, they offload a ton of management drudge, but it's an order of magnitude more fragile.

The unpredictability of performance, network contention and stability with EBS, for me, just doesn't outweigh the relatively smaller risk of hardware disk failure I take on from instance-backed nodes. Yes, I know, disks fail - but EBS disks fail a lot, and when they do, good luck fighting the herd to spin up more -- or crap, now, even getting web console access to understand what the hell is happening. That's the irony here - API access (including issuing more IPs!) is "throttled" at precisely the time when you need it most.

My advice? Instance-backed >= large, and roll your own failover/DR/load balancing. Go ahead and "plan for failure" - but do it old school: plan for the more likely case of simple h/w failure, not the EBS control plane and everything that depends on it.

Re: AWS outage summary

#17

Post by michaelkscott » Sat, Oct 27, 2012, 5:17 AM UTC

For anyone interested, here are the comments on the App engine outage: http://news.ycombinator.com/item?id=4704973

Re: AWS outage summary

#18

Post by Firehed » Sat, Oct 27, 2012, 5:20 AM UTC

sokoloff wrote: post #12

blaines wrote: post #7

Earlier quoted context omitted.

I don't necessarily agree with your first two sentences, but I definitely agree with the last one. I know they're smart. I do have some concerns that they're having too much downtime. If there's one small flaw in the system it seems that the whole thing begins to fail. If they fix the problem, and it impacts the larger system in some other unknown way, a different equally crippling issue could present itself in the f…

If there's one small flaw in the system it seems that the whole thing begins to fail. I wonder if there is selection bias underlying that judgment? Reading all of Amazon's post-mortems of big events, it does seem that the whole thing is fragile. What I suspect is more likely true is that AWS suffers thousands of small failures every month and most are contained as designed, with no(or minuscule) customer impact. It's…

> What I suspect is more likely true is that AWS suffers thousands of small failures every month and most are contained as designed, with no(or minuscule) customer impact.

Isn't that the whole point of moving to The Cloud? There's supposed to be some magical system in place such that hardware failures are routed around and don't interrupt service. Of course you can roll this yourself with your own hardware, but this is done for you.

It should be no small surprise that a system complicated enough to appear magical has some crazy complexity behind the scenes, and accidental dependencies can result in catastrophic failure.

Re: AWS outage summary

#19

Post by signifiers » Sat, Oct 27, 2012, 5:20 AM UTC

taligent wrote: post #13

mrkurt wrote: post #10

AWS sure does put out amazing post mortems. If only they'd make their status page more useful ...

How is the status page not useful ? It's so simple to understand. Green if everything is fine. Green if there are intermittent problems. Green if nothing works.

Too true. Today when Google's App Engine failed hard, they flat out said: "App Engine is currently experiencing serving issues – Python, Java, Go … Oct 26 2012, 07:30 AM-11:59 PM … Current Availability: 55.90%" and showed read/write I/O metrics were spiking through the roof. Their dashboard clearly labeled the relevant icon as the most severe "Service Interruption".

I admire that level of transparency.

Re: AWS outage summary

#20

Post by tonybaroneee » Sat, Oct 27, 2012, 5:22 AM UTC

taligent wrote: post #13

mrkurt wrote: post #10

AWS sure does put out amazing post mortems. If only they'd make their status page more useful ...

How is the status page not useful ? It's so simple to understand. Green if everything is fine. Green if there are intermittent problems. Green if nothing works.

Lol, thanks for a good laugh.