Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Post-Mortem

aws.amazon.com

Post reply on HN
Print view View original

41–50 of 69 posts

Re: AWS Post-Mortem

#41

Post by o1iver »

There seems to be a pretty simple solution to these problems: diversification. Like most things in life, putting all your eggs in one basket it not the right choice.

The people who use only AWS or only RackSpace or only 1&1 are equally wrong.

What you have to do it diversify. Run a production site ghost on some other platform (software/hardware bugs, ...), run by some other provider (bankruptcy, theft, ...) in another country (power cuts, earthquake, ...). As soon as the primary goes down you switch on the secondary. The probability of a total blackout is then squared: 10^-3*10^-3=10^-6.

The great thing with these "cloud" platforms is that your secondary system can even "go to sleep" saving you money and then spin up instances as soon as the primary goes down. This is by the way how banks, airport-systems and probably the NSA do it!

Re: AWS Post-Mortem

#42

Post by jwatte »

saetaes wrote: post #18
byoung2 wrote: post #7

Earlier quoted context omitted.

IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems That happened at Rackspace a few years back: http://techcrunch.com/2009/06/30/what-went-down-at-rackspace... I have an account with GoGrid, and they do a regular testing of their backup generators. I'm not sure if they throw the switch on the mains, though.

Monthly generator testing is, and should be, standard for any data center. Same with the UPSes - monthly testing to make sure they can handle the load long enough for the generators to kick in. Throwing the switch on the mains is probably not happening anywhere on a regular basis, though. There may be "routine" events (some sort of electrical infrastructure upgrade) that causes the data center to be put onto generato…

Why would throwing the switch be risky? It's supposed to be HA. If it doesn't work, that's a bug, and you fix it! Just like backups are not backups until they have been restored (we verify this by making our data warehouse depend on the backup) and hot standbys aren't standbys until switched in (we do this to databases regularly.) Netflix apparently has a chaos generator that randomly kills machines as a standard process. If you're supposed to deal with failure, make sure you're dealing with failure regularly!

Re: AWS Post-Mortem

#43

Post by jwatte »

rdl wrote: post #29

The thing I wonder about is wtf they didn't manually switch to generator when their automatic controls failed. They had presumably ~5 minutes of UPS; it took them 40 minutes to do this. This probably isn't directly Amazon's fault, but whatever contract datacenter they are using in Europe (probably a PTT, or possibly an international carrier; really curious what facility) I'm wary of using >1 generators to back up loa…

They explained that a ground fault prevented generators from delivering power. Manual start doesn't help in that case.

Re: AWS Post-Mortem

#44

Post by spartango »

larrycatinspace wrote: post #2

I'm thinking AWS needs to implement the Availability Zones: AZ-ChaosMonkey and AZ-ChaosApe. Having a dedicated playground for breaking things where they can start to observe how this complex system reacts to simple failures and gaps in assumptions.

This is in the pipe.

Re: AWS Post-Mortem

#45

Post by rdl »

jwatte wrote: post #43
rdl wrote: post #29

The thing I wonder about is wtf they didn't manually switch to generator when their automatic controls failed. They had presumably ~5 minutes of UPS; it took them 40 minutes to do this. This probably isn't directly Amazon's fault, but whatever contract datacenter they are using in Europe (probably a PTT, or possibly an international carrier; really curious what facility) I'm wary of using >1 generators to back up loa…

They explained that a ground fault prevented generators from delivering power. Manual start doesn't help in that case.

From what I read, they said ground fault confused their PLCs (synchro gear for paralleling multiple generators). This shouldn't affect the generator (engine, generator) outputting power.

Electronics are much more sensitive to ground faults, etc. than mechanical and electrical devices.

A big manual transfer switch (as backup), which is presumably what they ended up using, is fairly bulletproof.

Re: AWS Post-Mortem

#46

Post by dangrossman »

Locke1689 wrote: post #38
saturn wrote: post #21

Earlier quoted context omitted.

You mean availability zone. Well I guess me, Reddit, Foursquare, and plenty of other sites just got lucky in the bad availability zone. Ah yes, here's the classic AWS apologist pattern in full effect. You don't use EBS! Of course you don't, you would have to be some kind of friggin' idiot to use EBS. So what do you use for, say, MongoDB data files, that is different from morons like me who stupidly assumed they could…

S3?

How would you run your database off a file storage service?

Re: AWS Post-Mortem

#47

Post by dangrossman »

saturn wrote: post #6

As someone who has put a considerable amount of resources moving things into cloud computing - I wanted to believe. But I have changed my mind. Cloud computing scales the efficiencies, yes. It also scales the problems. And because of this, AWS is by several orders of magnitude the worst of my current hosts. I have dedicated servers. No downtime in past year. I have a couple of cloud servers with rackspace. No downtim…

Same story.

A few years back I had moved the majority of my sites onto Amazon's web services, using more and more of them as they were released. W3Counter.com was, just a year ago, using multiple EC2 instances of various sizes, Elastic Load Balancers, Amazon RDS instances with Multi-AZ failover, RAID arrays of EBS disks... and during those years, its reliability and performance degraded significantly compared to when it was run on physical servers. I was hit by the big Virginia outage as well.

Two months ago, I took the time to rearchitect everything and move every site back to various dedicated servers at SoftLayer. I feel like I'm in control again... no more worrying about things like the network latency of my hard disks. Despite prepaying significantly for reserved instances on both EC2 and RDS, the costs of dedicated hardware are still significantly less, too.

It was painful to waste thousands of dollars in reserved resources I'll now never use, but you know what they say about sunk costs.

Re: AWS Post-Mortem

#48

Post by cperciva »

wes-exp wrote: post #30
cperciva wrote: post #13

Earlier quoted context omitted.

One zone in US-East had a significant outage in April; and only instances using EBS were affected.

Actually some of the issues extended to the entire East region . In particular, for quite some time I couldn't create a new instance in any East zone. At all.

True, but "can't instantaneously spin up a new instance" just means that for a few hours the US-East EC2 region was providing the same amount of flexibility as dedicated servers.

Re: AWS Post-Mortem

#50

Post by marcamillion »

It seems to me that Amazon Web Services will never truly be VERY stable.

Not because I am being cynical, but just based on the nature of what they are doing.

They are the biggest provider of large scale cloud-based computing services. They are pushing the boundaries. They are bound to always come upon problems that no one has ever seen before (including themselves) just based on the very nature of their business.

So if you are looking for 'rock-solid reliability', maybe it is better to wait for another big company (Google, Apple, etc.) to come behind and fix all the mistakes that Amazon made the first time.

That being said, I use AWS and I love it. Granted, I don't use EBS (not directly, via Heroku) and yes I have encountered downtime recently, it's not that big of a deal. I know they aren't messing around, and they are in uncharted territory.

I can't reasonably expect them to have the best uptime for a platform that no one has ever built or done before, on the first time around the block. That's very unreasonable.

That being said, I will continue using them from now until I outgrow them or the economics becomes painful, because the value I get with paying for what I use far outweighs 24 - 48 hours of downtime per year.

Post reply on HN