Amazon Web Services are down • Hacker News Board

Re: Amazon Web Services are down

#301

Post by hoop » Thu, Apr 21, 2011, 5:47 PM UTC

jerf wrote: post #260

hoop wrote: post #216

Earlier quoted context omitted.

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Good point. let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one regio…

"let's assume (and hope) the probability of failure of one region is independent of the state of the other regions." In practice, that's not true, and it's not true enough to ruin the entire rest of your calculations. For Amazon regions to function independently, they'd have to be actually, factually independent ; there is no interaction between them. The reaction to one node going down is never to increase the load…

I agree with you 100% that they're not independent, but I don't know enough about the data to model the probabilities of failure and availability in a HN comment :-)

After today's event, it would certainly be interesting to see how resource consumption changed in other availability zones and at other providers during this outage.

I wonder if that could be measured passively? What I mean is, by monitoring response times of various services that are known to be in specific regions and seeing how that metric changes (as opposed to waiting on a party that has little-to-no economic benefit to release that information.)

Re: Amazon Web Services are down

#302

Post by jen_h » Thu, Apr 21, 2011, 5:54 PM UTC

mtodd wrote: post #283

Why is ELB not mentioned at all on the Service Health Dashboard? We're experiencing problems with two of our ELBs, one indicating instance health as out of service, reporting "a transient error occurred". Another, new LB (what we hoped would replace the first problematic LB), reports: "instance registration is still in progress". A support issue with Amazon indicated that it was related to the ongoing issues and to m…

We've got a single non-responsive load balancer IP in one of our primary ELBs (others have been fine for several hours now), so while everything else for us is up & running, still have transient errors for folks that get shunted to through that one system.

The interesting thing about the ELB in a situation like this is that I believe it may, in many instances, be better to hobble along and deal with an elevated error rate if at least some of your ELB hosts are working than to re-create the entire ELB somewhere else, especially if you're a high-traffic site where you may hit scaling issues going from 0 to 60 in milliseconds (OMMV, but we've been spooked enough in the past not to try anything hasty until things get back to normal).

Re: Amazon Web Services are down

#303

Post by jjm » Thu, Apr 21, 2011, 5:55 PM UTC

jedberg wrote: post #238

bvi wrote: post #225

Earlier quoted context omitted.

You guys might have answered this in one of your AMAs/blog posts (or was it raldi who commented?), but what options can reddit resort to should this stuff happen again to this degree of severity?

We're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option. So I guess that is the long way of saying that hopefully it won't happen again.

I do not believe you could be effective by moving away from EBS, you know without giving up quite a bit.

Doing things the right way with EC2 means using EBS. It's the brake caliper to the rotor. Sure you could have drum brakes but they're not nearly anywhere effective as they quickly get heat soaked. I'm referring to S3.

One should trust ephemeral storage. Your instance can go down at anytime. Write speeds to S3 are not nearly as fast as ephemeral or EBS arrays (raid).

Hate to say it, but If one cannot trust EBS then what the heck are 'we' doing on EC2... EBS quality should be priority one, otherwise we're all building Skyscrapers on foam foundations of candy cane rebar.

Re: Amazon Web Services are down

#304

Post by shasta » Thu, Apr 21, 2011, 5:58 PM UTC

ry0ohki wrote: post #256

BrandonM wrote: post #243

Earlier quoted context omitted.

Why does Reddit really need 99% availability? Is a customer unduly harmed or is the world even worse off if Reddit is down for a couple cumulative days per year? Is it worth the cost? Would you put up with more ads and/or pay for Reddit in order to make sure that it's available 24/7/365?

Probably not as much for the customer as for the company. When sites are unreliable, people end up going to the more reliable competitors as they arise.

I wouldn't think there a large number of customers deciding "this is too unreliable, I'm leaving" on the basis of a few hours of down time. On the other hand, there might be a large number of people who, upon finding your site down, decide to visit alternatives that are up at the time, and some of those people might decide they like the alternatives better.

Re: Amazon Web Services are down

#305

Post by jjm » Thu, Apr 21, 2011, 5:58 PM UTC

joshhart wrote: post #296

epi0Bauqu wrote: post #274

Earlier quoted context omitted.

How do you move away from EBS and still deal with large data?

I imagine they'd do consider some combination of the following (sorted by most likely) 1. Sharding data 2. Pulling tables out to other servers from the main DB 3. Pruning excessive data 4. Compressing data

It still has to be stored somewhere though right? If it's EBS you've just made yourself a complicated solution that will eventually fail all over again. No?

Re: Amazon Web Services are down

#306

Post by bdonlan » Thu, Apr 21, 2011, 6:05 PM UTC

asymptotic wrote: post #129

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Amazon's EBS SLA is less clear, but they state that they expect an annual failure rate of 0.1-0.5%, compared to…

Note also that 0.1-0.5% refers to irrecoverable data loss, not temporary unavailability.

Re: Amazon Web Services are down

#307

Post by jjm » Thu, Apr 21, 2011, 6:12 PM UTC

Everyone talks about SLAs but I believe it doesn't consider the fact that the EBS vols are still up (not on fire, and available) and are phantom writing or that the network is queued up the wazoo so writes don't even happen in a timely manner as you'd expect.

Re: Amazon Web Services are down

#308

Post by jjm » Thu, Apr 21, 2011, 6:18 PM UTC

prakash wrote: post #285

timf wrote: post #222

Some quotes regarding how Netflix handled this without interruptions: "Netflix showed some increased latency, internal alarms went off but hasn't had a service outage." [1] "Netflix is deployed in three zones, sized to lose one and keep going. Cheaper than cost of being down." [2] [1] https://twitter.com/adrianco/status/61075904847282177 [2] https://twitter.com/adrianco/status/61076362680745984

Here's the 24h latency data on EC2 east, west, eu, apac: http://dl.dropbox.com/u/1898990/EC2-multiple-zones-24h.png Last 60 minutes comparison data: http://dl.dropbox.com/u/1898990/EC2-multiple-zones-60m.png time in GMT. A study we (Cedexis) did in January comparing multiple ec2 zones and other cloud providers: (pdf) http://dl.dropbox.com/u/1898990/76-marty-kagan.pdf

Pure opinion: That convergence might show that Amazon tried to do a failover on a DC level. Once they figured that wouldn't work or east was down for the count they just let it cycle to the ground under latency.

Re: Amazon Web Services are down

#309

Post by spydez » Thu, Apr 21, 2011, 6:21 PM UTC

endgame wrote: post #164

kristianp wrote: post #139

Earlier quoted context omitted.

Actually the official line is that Easter Monday got moved to Tuesday.

Interesting. Thanks. What did the Catholic Church have to say about that? Is it that Easter Monday is still on Monday, but the holiday is on Tuesday?

Easter is the holiday, and it's on Sunday. I'm pretty sure the Pope doesn't care much what people do the day after Easter (or the day after that).

Re: Amazon Web Services are down

#310

Post by elliottcarlson » Thu, Apr 21, 2011, 6:24 PM UTC

jjm wrote: post #307

Everyone talks about SLAs but I believe it doesn't consider the fact that the EBS vols are still up (not on fire, and available) and are phantom writing or that the network is queued up the wazoo so writes don't even happen in a timely manner as you'd expect.

I'm not sure that just because they are up, yet unusable, would negate an SLA.

You could have a dedicated server in a datacenter - if the network goes out, your machine is still up and happily waiting to serve requests - but it's still unusable and not actually in service.