Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Post-Mortem

aws.amazon.com

Post reply on HN
Print view View original

61–69 of 69 posts

Re: AWS Post-Mortem

#61

Post by derefr »

streptomycin wrote: post #60
tednaleid wrote: post #32

Earlier quoted context omitted.

We use EBS, and had machines in the availability zone that went down that were affected. Those machines were out for longer than a day, but we were back up within an hour because we had redundancies built in across other availability zones. If you're doing anything that matters, you can't rely on a single zone/machine/whatever, no matter who your hosting provider is.

So you're saying we need a cloud of clouds?

Just a higher level of abstraction. Many people don't want to care about what zone they're running in, or really anything about the machine (or even virtual machine) they're running on; they just want to run their apps "somewhere" that persists through hardware problems. Maintaining such a thing isn't Amazon's charter, though; it's more the job of a cloud application hosting provider, like Heroku (who I'm surprised isn't multi-homing their apps on several clouds by now for just this reason.)

Re: AWS Post-Mortem

#62

Post by wes-exp »

Locke1689 wrote: post #55
lmz wrote: post #53

Earlier quoted context omitted.

It's the root device that's on EBS. On most AMIs with EBS root that's the entire OS i.e., /etc, /dev, /bin, /usr, etc. and not just the OS kernel.

And for the things that matter (i.e., not fake mount points) that will all mostly live in resident memory.

... and your point is?

Re: AWS Post-Mortem

#63

Post by Kadin »

jwatte wrote: post #42
saetaes wrote: post #18

Earlier quoted context omitted.

Monthly generator testing is, and should be, standard for any data center. Same with the UPSes - monthly testing to make sure they can handle the load long enough for the generators to kick in. Throwing the switch on the mains is probably not happening anywhere on a regular basis, though. There may be "routine" events (some sort of electrical infrastructure upgrade) that causes the data center to be put onto generato…

Why would throwing the switch be risky? It's supposed to be HA. If it doesn't work, that's a bug, and you fix it! Just like backups are not backups until they have been restored (we verify this by making our data warehouse depend on the backup) and hot standbys aren't standbys until switched in (we do this to databases regularly.) Netflix apparently has a chaos generator that randomly kills machines as a standard pro…

> Netflix apparently has a chaos generator that randomly kills machines as a standard process.

This sounds pretty neat, but a quick Google didn't turn up any information about it besides this post. Do you know of anywhere to get more information on what they're doing? It sounds like a sensible idea, although I can only imagine trying to implement it would be ... challenging, for most companies/organizations.

Re: AWS Post-Mortem

#64

Post by dreww »

Kadin wrote: post #63
jwatte wrote: post #42

Earlier quoted context omitted.

Why would throwing the switch be risky? It's supposed to be HA. If it doesn't work, that's a bug, and you fix it! Just like backups are not backups until they have been restored (we verify this by making our data warehouse depend on the backup) and hot standbys aren't standbys until switched in (we do this to databases regularly.) Netflix apparently has a chaos generator that randomly kills machines as a standard pro…

> Netflix apparently has a chaos generator that randomly kills machines as a standard process. This sounds pretty neat, but a quick Google didn't turn up any information about it besides this post. Do you know of anywhere to get more information on what they're doing? It sounds like a sensible idea, although I can only imagine trying to implement it would be ... challenging, for most companies/organizations.

check out item no 3. on this list, which is AWS lessons-learned: http://techblog.netflix.com/2010/12/5-lessons-weve-learned-u...

see also: http://techblog.netflix.com/2011/04/lessons-netflix-learned-...

and: http://techblog.netflix.com/2011/07/netflix-simian-army.html for the other simian themed services they've developed for care and feeding of their AWS stuff.

Re: AWS Post-Mortem

#65

Post by ericd »

streptomycin wrote: post #60
tednaleid wrote: post #32

Earlier quoted context omitted.

We use EBS, and had machines in the availability zone that went down that were affected. Those machines were out for longer than a day, but we were back up within an hour because we had redundancies built in across other availability zones. If you're doing anything that matters, you can't rely on a single zone/machine/whatever, no matter who your hosting provider is.

So you're saying we need a cloud of clouds?

Sounds like a certain exotic financial instrument...

Re: AWS Post-Mortem

#66

Post by robryan »

This seems to share a lot of parallels with the last big outage in terms of the API request overload and the EBS replication. Seems like the system need to be able to tell a bit better between a node going down and require a remirror and most of an availability zone going down.

Re: AWS Post-Mortem

#67

Post by generators »

saturn wrote: post #6

As someone who has put a considerable amount of resources moving things into cloud computing - I wanted to believe. But I have changed my mind. Cloud computing scales the efficiencies, yes. It also scales the problems. And because of this, AWS is by several orders of magnitude the worst of my current hosts. I have dedicated servers. No downtime in past year. I have a couple of cloud servers with rackspace. No downtim…

why not try google app engine ? its reliability increase many folds since its start ( hardly an hour downtime in my app in HR datastore...).

Re: AWS Post-Mortem

#68

Post by Kartick »

Isn't the real problem attempting to build an abstraction that lets programmers assume the datacenter won't go down? Aren't you better off building a system like Google's Megastore/BigTable that assumes datacenters go down, and handling it in software?

The obvious downside is, of course, that you'll have to rewrite all your software, but doesn't that give higher reliability at the end of the day?

Disclosure: I work for Google, but on stuff totally disconnected with any of this.

Re: AWS Post-Mortem

#69

Post by rapind »

aaronbrethorst wrote: post #57
rapind wrote: post #33

Earlier quoted context omitted.

I'm the same. I fully embraced S3 and EC2 when they came out (even played with SQS) and enthusiastically told everyone I could that this was the future, it's the new electricity, etc. While I still think that eventually it will end up as a utility I'm opting out of the cloud for anything production for the time being. I'll keep an eye on it of course. Does anyone know if Heroku has spread their services across HA zon…

Does anyone know if Heroku has spread their services across HA zones? I don't know for a fact, but the multiple Heroku sites of mine that went down concurrently with this problem would indicate 'no.'

Good to know, thanks. I'll have to check into that for one of my projects.
Post reply on HN