Amazon Web Services are down • Hacker News Board

Re: Amazon Web Services are down

#191

Post by ceejayoz » Thu, Apr 21, 2011, 1:58 PM UTC

hendi_ wrote: post #45

jedberg wrote: post #24

Earlier quoted context omitted.

That's like saying "yay for relying on a datacenter"

Note that I've written "relying", not "hosting", meaning I'm well aware of the fact that using "the cloud" as an additional layer of capacity can be a good thing. But relying on the cloud means that you're giving up control. Power outage in your data center? Buy diesel generators next time! Faulty network card? Put in two redundant ones! AWS down? Wait and pray!

> Power outage in your data center? Buy diesel generators next time!

> AWS down? Wait and pray!

Wouldn't you be waiting and praying if your datacenter lost power and didn't have diesel generators, too?

Re: Amazon Web Services are down

#192

Post by [deleted] » Thu, Apr 21, 2011, 1:58 PM UTC

ghshephard wrote: post #115

olegp wrote: post #101

Assuming the problem is indeed with EBS, I would say this should be a warning sign to anyone considering going with a PaaS provider, which Amazon is quickly becoming, instead of an IaaS provider like Slicehost or Linode. The increased complexity of their offering makes it more likely that things will break, leaving you locked in. I did a 15 minute talk on the subject, which you can check out here: http://iforum.com.u…

Every time someone makes the claim that downtime should be a warning sign about going with a PaaS provider (or, indeed, an IaaS provider, or in some cases, people even make this claim about going with someone else's Data Center) - I always respond: "And why do you believe that you would do any better?" Every environment I've been involved in as an operations professional for the last 15 years has experienced downtime…

[deleted]

Re: Amazon Web Services are down

#193

Post by tedjdziuba » Thu, Apr 21, 2011, 1:59 PM UTC

stef25 wrote: post #132

asymptotic wrote: post #129

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Amazon's EBS SLA is less clear, but they state that they expect an annual failure rate of 0.1-0.5%, compared to…

Reddit's been down for several hours today, I'm sure they are already way lower 99.95%.

Reddit being down is not news.

Re: Amazon Web Services are down

#194

Post by bad_user » Thu, Apr 21, 2011, 1:59 PM UTC

antonioe wrote: post #145

yuvadam wrote: post #2

Current status: bad things are happening in the North Virginia datacenter. EC2, EBS and RDS are all down on US-east-1. Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

Foursquare is up now. Quora not showing the 503 anymore, Reddit still down.

Foursquare is up, but Quora is still showing the 503.

Re: Amazon Web Services are down

#195

Post by tedjdziuba » Thu, Apr 21, 2011, 2:02 PM UTC

paraschopra wrote: post #183

http://venuetastic.com/ - feel bad for these guys. They launched yesterday and down today because of AWS. Murphy's law in practice.

They are scaling in the cloud, at least.

Re: Amazon Web Services are down

#196

Post by retube » Thu, Apr 21, 2011, 2:02 PM UTC

retube wrote: post #90

thirtysixred wrote: post #86

Earlier quoted context omitted.

SHUT. DOWN. EVERYTHING. Good plan though. :p

Reminds me of that scene in Jurassic Park

Good lord. What's with all the DVs??

Re: Amazon Web Services are down

#197

Post by jedberg » Thu, Apr 21, 2011, 2:05 PM UTC

jbellis wrote: post #184

jedberg wrote: post #157

Earlier quoted context omitted.

> If the Reddit web server admins took availability seriously We do. > they would have chosen to deploy across more than one region. It's far too costly to do that. We are deployed across multiple AZs, but this failure hit multiple AZs.

Why is it more expensive to deploy in zones X,Y in regions A,B than zones M,N in region C? I assume you don't just mean "US West is ~10% more expensive than US East."

It's the combination of the extra cost of having machines in US West plus the cost of keeping the data synchronized between them (which is a lot) plus the added development overhead of making sure that things work cross region.

We'll get there one day, but we aren't there yet.

Re: Amazon Web Services are down

#198

Post by trotsky » Thu, Apr 21, 2011, 2:07 PM UTC

dspillett wrote: post #154

asymptotic wrote: post #136

Earlier quoted context omitted.

If the Reddit web server admins took availability seriously they would have chosen to deploy across more than one region. Do you disagree? Why do you disagree? I'm being honest, no snark involved in my questions.

He wasn't suggesting that all Reddit's problems are due to Amazon services, he was using Reddit's down time today as a data point illustrating that the uptime guarantee claimed for the service has not been kept this year (in fact a whole year's "permitted downtime" as implied by the 99.95% SLA may be eaten on one day). Presumably Amazon will be handing out some refunds and other compensation (assuming the SLA isn't o…

Very few cloud SLA's seem to have teeth to me. Amazon's SLA gives service credit equal to 10% of your total bill for the billing period if they blow past the 0.05%. This is a lot better than some cloud providers that will simply prorate the downtime, but pretty crappy in terms of actual business compensation. It's equivalent to a sales discount almost any organization with a sales staff could write without thinking about it - meaning Amazon is still making money on every customer even when they've blown past their SLA - assuming every single customer fills out the forms to apply for the discount. Hint: Many won't, see mail in rebates.

A number of tier 1 network providers offer certain customers SLA's that are clearly in place to prove that they invest in redundancy and disaster planning. ex: less than 99.99% --> 10% credit. less than 99.90% --> no charges for the circuit in the billing period.

This reflects an understanding that downtime can hurt your business/infrastructure far in excess of the measurable percentage.

Re: Amazon Web Services are down

#199

Post by tybris » Thu, Apr 21, 2011, 2:07 PM UTC

All hosting services go down occasionally. If you want to stay up you need to build a fault-tolerant distributed system that spans multiple regions and potentially multiple providers.

Also, Amazon should fix EBS.

Re: Amazon Web Services are down

#200

Post by chrislomax » Thu, Apr 21, 2011, 2:08 PM UTC

jedberg wrote: post #180

chrislomax wrote: post #118

Earlier quoted context omitted.

I thought you could snapshot drives across regions and bring those EBS drives up under new instances in new regions? I have not used it all in detail yet so I don't know the practicality of this method. I think I will stick to my co-location costings I am doing for the time being. There is only one person to rely on when it all goes wrong then! Good luck getting it sorted, i know I wouldn't appreciate being up at 3am…

> I thought you could snapshot drives across regions and bring those EBS drives up under new instances in new regions? In theory, yes. In practice, those snapshots hurt the volume so much that it is impossible to take one in production.

Interesting, your insight has given me a lot to think about.

Do you guys blog this anywhere?