Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Post-Mortem

aws.amazon.com

Post reply on HN
Print view View original

11–20 of 69 posts

Re: AWS Post-Mortem

#11

Post by saturn »

cperciva wrote: post #9
saturn wrote: post #6

As someone who has put a considerable amount of resources moving things into cloud computing - I wanted to believe. But I have changed my mind. Cloud computing scales the efficiencies, yes. It also scales the problems. And because of this, AWS is by several orders of magnitude the worst of my current hosts. I have dedicated servers. No downtime in past year. I have a couple of cloud servers with rackspace. No downtim…

AWS? More than 24hrs downtime in the last year. You must have incredible luck. Sure, AWS has a lot of outages; but most of them only affect a small proportion of their users.

Huh? North Virginia was down for something like 30 hours in April. I had some instances in there. Hardly "incredible luck". And it was not just me: http://www.google.com.au/search?q=AWS+Virginia+outage

Re: AWS Post-Mortem

#12

Post by cperciva »

harshaw wrote: post #4
larrycatinspace wrote: post #2

I'm thinking AWS needs to implement the Availability Zones: AZ-ChaosMonkey and AZ-ChaosApe. Having a dedicated playground for breaking things where they can start to observe how this complex system reacts to simple failures and gaps in assumptions.

Sure. Presumably Amazon has a test lab that replicates multiple zones :) Perhaps your point is that Amazon should make this test lab public so people can contribute to the QA effort? IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems (I'm thinking of some past failures at softlayer and other providers). I wonder if it is prohibitively…

I wonder if it is prohibitively expensive to do a real life system test on a big data center

It's probably prohibitively dangerous. Backup power systems don't have many-nines of reliability; generators which are reliable enough for the once-a-decade event when a car crash knocks out your utility power aren't anywhere near the reliability needed to run your datacentre for an hour every month as a test.

Re: AWS Post-Mortem

#13

Post by cperciva »

saturn wrote: post #11
cperciva wrote: post #9

Earlier quoted context omitted.

AWS? More than 24hrs downtime in the last year. You must have incredible luck. Sure, AWS has a lot of outages; but most of them only affect a small proportion of their users.

Huh? North Virginia was down for something like 30 hours in April. I had some instances in there. Hardly "incredible luck". And it was not just me: http://www.google.com.au/search?q=AWS+Virginia+outage

One zone in US-East had a significant outage in April; and only instances using EBS were affected.

Re: AWS Post-Mortem

#14

Post by saturn »

cperciva wrote: post #13
saturn wrote: post #11

Earlier quoted context omitted.

Huh? North Virginia was down for something like 30 hours in April. I had some instances in there. Hardly "incredible luck". And it was not just me: http://www.google.com.au/search?q=AWS+Virginia+outage

One zone in US-East had a significant outage in April; and only instances using EBS were affected.

The cheapest zone, yes, so presumably the most popular. And who doesn't use EBS?

You sound like you think I'm being unfair. Why? Plenty of sites were badly affected by that outage; it precipitated a lot of self-examination at some companies I know of. I don't think I am overstating the issues here.

Re: AWS Post-Mortem

#15

Post by karambahh »

saturn wrote: post #6

As someone who has put a considerable amount of resources moving things into cloud computing - I wanted to believe. But I have changed my mind. Cloud computing scales the efficiencies, yes. It also scales the problems. And because of this, AWS is by several orders of magnitude the worst of my current hosts. I have dedicated servers. No downtime in past year. I have a couple of cloud servers with rackspace. No downtim…

I think you nailed it. To each his own, and a good infrastructure probably is a mix of cloud, virtual, physical servers. I currently have a similar setup as yours (but I suspect mine is smaller, about 40 machines total), but my main concern is with my primary DC which seems to periodically loose utility power, loose A/C, cut all fibers at once, etc....

Re: AWS Post-Mortem

#16

Post by cperciva »

saturn wrote: post #14
cperciva wrote: post #13

Earlier quoted context omitted.

One zone in US-East had a significant outage in April; and only instances using EBS were affected.

The cheapest zone, yes, so presumably the most popular. And who doesn't use EBS? You sound like you think I'm being unfair. Why? Plenty of sites were badly affected by that outage; it precipitated a lot of self-examination at some companies I know of. I don't think I am overstating the issues here.

The four zones in US-East have exactly the same prices.

And I don't use EBS.

Re: AWS Post-Mortem

#17

Post by tlack »

quanticle wrote: post #8
saturn wrote: post #6

As someone who has put a considerable amount of resources moving things into cloud computing - I wanted to believe. But I have changed my mind. Cloud computing scales the efficiencies, yes. It also scales the problems. And because of this, AWS is by several orders of magnitude the worst of my current hosts. I have dedicated servers. No downtime in past year. I have a couple of cloud servers with rackspace. No downtim…

Is there any reason in particular why you wouldn't recommend Rackspace?

For what its worth, I found Rackspace's support to not be worth the extremely high cost. Softlayer is just as good.

Re: AWS Post-Mortem

#18

Post by saetaes »

byoung2 wrote: post #7
harshaw wrote: post #4

Earlier quoted context omitted.

Sure. Presumably Amazon has a test lab that replicates multiple zones :) Perhaps your point is that Amazon should make this test lab public so people can contribute to the QA effort? IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems (I'm thinking of some past failures at softlayer and other providers). I wonder if it is prohibitively…

IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems That happened at Rackspace a few years back: http://techcrunch.com/2009/06/30/what-went-down-at-rackspace... I have an account with GoGrid, and they do a regular testing of their backup generators. I'm not sure if they throw the switch on the mains, though.

Monthly generator testing is, and should be, standard for any data center. Same with the UPSes - monthly testing to make sure they can handle the load long enough for the generators to kick in. Throwing the switch on the mains is probably not happening anywhere on a regular basis, though. There may be "routine" events (some sort of electrical infrastructure upgrade) that causes the data center to be put onto generator power, but throwing the mains just to test is a very risky endeavor, and one that a data center provider who has very high power availability guarantees with expensive penalties, is not like likely to undertake.

Re: AWS Post-Mortem

#19

Post by ams6110 »

cperciva wrote: post #12
harshaw wrote: post #4

Earlier quoted context omitted.

Sure. Presumably Amazon has a test lab that replicates multiple zones :) Perhaps your point is that Amazon should make this test lab public so people can contribute to the QA effort? IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems (I'm thinking of some past failures at softlayer and other providers). I wonder if it is prohibitively…

I wonder if it is prohibitively expensive to do a real life system test on a big data center It's probably prohibitively dangerous. Backup power systems don't have many-nines of reliability; generators which are reliable enough for the once-a-decade event when a car crash knocks out your utility power aren't anywhere near the reliability needed to run your datacentre for an hour every month as a test.

On the other hand, just as with database backups, making them is only half of the story. You have to test restores/recovery. Does your plan actually work? What have you overlooked? What edge cases do you need to accommodate?

Many data centers will test backup power generation regularly just for this reason. It's not unheard of at all and the risk of a problem at a planned time is worth the confidence in knowing that the system is more likely to work when needed at an unexpected time.

Re: AWS Post-Mortem

#20

Post by robszumski »

saturn wrote: post #10
quanticle wrote: post #8

Earlier quoted context omitted.

Is there any reason in particular why you wouldn't recommend Rackspace?

Hm. Well, I don't like them. It's subjective, you might disagree. But off the top of my head: 1. Contracts. They want 1 years minimum contracts for any dedicated servers. For truly gargantuan orders I could understand this but for one puny server? Never. 2. Their definition of "cloud" is different from mine. To use their "cloud" services your servers need to be public facing, ie on public IPs. Want them on your own V…

It sounds like you're not a fan of Rackspace (and that's fine) but you can't honestly believe that an API sends an email to a tech. That's just completely false statement that no one in there right mind should believe.
Post reply on HN