Amazon Web Services are down • Hacker News Board

Re: Amazon Web Services are down

#251

Post by cube13 » Thu, Apr 21, 2011, 3:16 PM UTC

forgotusername wrote: post #201

cmurdock wrote: post #153

Earlier quoted context omitted.

Reddit experienced some issues with Amazon a month ago that resulted in the site being down for almost a day. I'm pretty sure they're way below that percentage.

That's conflating SLAs again, Reddit's long-running problems have been with EBS reliability.

Even so, .5% downtime per year is about 44 hours, and Reddit's definitely had more downtime in the last few months than that.

Of course, that's assuming that 100% of Reddit's problems were due to EBS only and not a combination of EBS, EC2 and their own code.

Re: Amazon Web Services are down

#252

Post by random42 » Thu, Apr 21, 2011, 3:18 PM UTC

helium wrote: post #205

random42 wrote: post #171

Earlier quoted context omitted.

Actually, I'd expect Heroku to not charge for when the site was down, as they are clearly not available, it does not sound fair if they charge for it. Am I expecting too much from them?

My app on heroku is running, it's just that I can't log in to their management console to de-allocate resources that I am paying for by the hour.

If I were you, I'd send them an email requesting this, on your behalf. At the end of the day, its their responsibility to make the console unavailable. I will be more than unimpressed if they dont see this logic here.

Re: Amazon Web Services are down

#253

Post by wallnutboy » Thu, Apr 21, 2011, 3:18 PM UTC

jedberg wrote: post #20

Joakal wrote: post #18

Earlier quoted context omitted.

How does Reddit display the 'offline' page if it's down?

The server is still up, so we can serve it right out of the load balancer.

Are you able to enable the 'read-only-mode' using the same method?

Re: Amazon Web Services are down

#254

Post by agotterer » Thu, Apr 21, 2011, 3:20 PM UTC

wmoxam wrote: post #32

yuvadam wrote: post #2

Current status: bad things are happening in the North Virginia datacenter. EC2, EBS and RDS are all down on US-east-1. Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

Not all EC2 & EBS instances are down. I have several in US-east-1a and 1 is down, while all of the others are working.

We had about 45 min of downtime around 4am EST. Our RDS instances, EBS backed and normal instances all returned without problems. We are in Virginia us-east-1a and us-east-1b.

Re: Amazon Web Services are down

#255

Post by gairbheil » Thu, Apr 21, 2011, 3:22 PM UTC

hoop wrote: post #216

asymptotic wrote: post #129

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Amazon's EBS SLA is less clear, but they state that they expect an annual failure rate of 0.1-0.5%, compared to…

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Good point. let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one regio…

No, your decimal point is off. 0.05% * 0.05% = 0.0005 * 0.0005 = 0.00000025, or 0.000025%. It works out to an expected downtime of 8 seconds per year, instead of over 4 hours for one location.

Of course, redundancy doesn't set itself up, so there are added costs on top of Amazon.

Re: Amazon Web Services are down

#256

Post by ry0ohki » Thu, Apr 21, 2011, 3:24 PM UTC

BrandonM wrote: post #243

asymptotic wrote: post #136

Earlier quoted context omitted.

If the Reddit web server admins took availability seriously they would have chosen to deploy across more than one region. Do you disagree? Why do you disagree? I'm being honest, no snark involved in my questions.

Why does Reddit really need 99% availability? Is a customer unduly harmed or is the world even worse off if Reddit is down for a couple cumulative days per year? Is it worth the cost? Would you put up with more ads and/or pay for Reddit in order to make sure that it's available 24/7/365?

Probably not as much for the customer as for the company. When sites are unreliable, people end up going to the more reliable competitors as they arise.

Re: Amazon Web Services are down

#257

Post by greaterscope » Thu, Apr 21, 2011, 3:31 PM UTC

Wish were able to download our ebs snapshots, which are supposedly hosted on S3. What does everyone else do?

Re: Amazon Web Services are down

#258

Post by singlow » Thu, Apr 21, 2011, 3:35 PM UTC

It's definitely a limited outage. My three instances seem to have operated all night with no problem. Two of them are EBS instances.

Re: Amazon Web Services are down

#259

Post by yatima2975 » Thu, Apr 21, 2011, 3:36 PM UTC

hoop wrote: post #216

asymptotic wrote: post #129

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Amazon's EBS SLA is less clear, but they state that they expect an annual failure rate of 0.1-0.5%, compared to…

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Good point. let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one regio…

Minor nitpick, but the availability should be even better, since 1% * 1% = 0.01% the availability becomes 99.999975% - six nines, anyone?

Re: Amazon Web Services are down

#260

Post by jerf » Thu, Apr 21, 2011, 3:37 PM UTC

hoop wrote: post #216

asymptotic wrote: post #129

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Amazon's EBS SLA is less clear, but they state that they expect an annual failure rate of 0.1-0.5%, compared to…

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Good point. let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one regio…

"let's assume (and hope) the probability of failure of one region is independent of the state of the other regions."

In practice, that's not true, and it's not true enough to ruin the entire rest of your calculations. For Amazon regions to function independently, they'd have to be actually, factually independent; there is no interaction between them. The reaction to one node going down is never to increase the load on other nodes as people migrate services, etc. There's fundamentally nothing you can do about the fact that if enough of your capacity goes out then you will experience demand in excess of supply.

If you want true redundancy you will at the very least need to go to another entirely separate service that is not Amazon... and if enough people do that, they'll break the effective independence of that arrangement, too.

(This is a special case of a more general rule, which is that computers are generally so reliable that the ways in which their probabilities deviate from Gaussian or independence tends to dominate your worst-case calculations.)