AWS Post-Mortem

I'm thinking AWS needs to implement the Availability Zones: AZ-ChaosMonkey and AZ-ChaosApe. Having a dedicated playground for breaking things where they can start to observe how this complex system reacts to simple failures and gaps in assumptions.

Sure. Presumably Amazon has a test lab that replicates multiple zones :) Perhaps your point is that Amazon should make this test lab public so people can contribute to the QA effort?

IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems (I'm thinking of some past failures at softlayer and other providers). I wonder if it is prohibitively expensive to do a real life system test on a big data center (or prohibitively expensive once the data center is on line). For example, how often do they turn off one of the mains (unexpectedly) to see what happens with the backup system?

As someone who has put a considerable amount of resources moving things into cloud computing - I wanted to believe. But I have changed my mind.

Cloud computing scales the efficiencies, yes. It also scales the problems. And because of this, AWS is by several orders of magnitude the worst of my current hosts.

I have dedicated servers. No downtime in past year. I have a couple of cloud servers with rackspace. No downtime (although i don't recommend them). I have some VPSes with local providers. No downtime.

AWS? More than 24hrs downtime in the last year. Seriously, for someone trying to run web sites reliably - screw that. I'm not using AWS any more.

And don't even get me started on the apologists. "EBS slow as treacle? Well you should have been running a multi zone raid-20 redundant array! Duh!". "EC2 instances dying at random? Well you should architect and implement a multi-master failover intelligent grid!"

I used to be under some kind of crazy delusional spell that the above was correct and it was somehow my fault that I wasn't correctly adapting to AWS's numerous failings. Well, no more. Now I realise that I should just stick with the super reliable service I know and love from traditional operators. You need to programmatically grow and shrink your app server flock? Great, use AWS. For the other 99.999% of us - stick with what you were using before.

Is there any reason in particular why you wouldn't recommend Rackspace?

Hm. Well, I don't like them. It's subjective, you might disagree. But off the top of my head:

1. Contracts. They want 1 years minimum contracts for any dedicated servers. For truly gargantuan orders I could understand this but for one puny server? Never.

2. Their definition of "cloud" is different from mine. To use their "cloud" services your servers need to be public facing, ie on public IPs. Want them on your own VPN? You can stil get the cloud prices but not the API. you create and cancel servers via tickets. This is different from VPS how?

3. Sloooow provisioning - even if you are able to use their "public cloud" API to provision a server - prepare to wait hours for it to be done, leading me to suspect it does nothing more than email a tech to provision a VPS and hook it up somehow. Oh, you can't pause them to save money either, again making me think these "cloud" servers are nothing more than slicehosts with an extra layer of abstraction

Is that enough? I could go on.