Amazon Web Services are down • Hacker News Board

Re: Amazon Web Services are down

#311

Post by mtodd » Thu, Apr 21, 2011, 6:26 PM UTC

jen_h wrote: post #302

mtodd wrote: post #283

Why is ELB not mentioned at all on the Service Health Dashboard? We're experiencing problems with two of our ELBs, one indicating instance health as out of service, reporting "a transient error occurred". Another, new LB (what we hoped would replace the first problematic LB), reports: "instance registration is still in progress". A support issue with Amazon indicated that it was related to the ongoing issues and to m…

We've got a single non-responsive load balancer IP in one of our primary ELBs (others have been fine for several hours now), so while everything else for us is up & running, still have transient errors for folks that get shunted to through that one system. The interesting thing about the ELB in a situation like this is that I believe it may, in many instances, be better to hobble along and deal with an elevated error…

We have an identical load balancer to one that is causing problems so we're lucky enough to reroute traffic through that one instead to get to the same boxes. (The boxes serve two different APIs through two different DNS CNAMEs so we split the ELBs for future and sanity). In this case, it's helped us out. Alternatively, we would've just routed all traffic to our west coast ELBs.

Re: Amazon Web Services are down

#312

Post by stevepotter » Thu, Apr 21, 2011, 6:44 PM UTC

potomak wrote: post #27

Quora says: "We'd point fingers, but we wouldn't be where we are today without EC2."

Yes, that was by far my favorite comment. Well played.

Re: Amazon Web Services are down

#313

Post by mattdeboard » Thu, Apr 21, 2011, 6:46 PM UTC

jasonlotito wrote: post #282

mattdeboard wrote: post #266

Earlier quoted context omitted.

I can't imagine how frustrating the jobs of the Reddit admins must be.

Is admin supposed to be plural? I mean, do they really have multiple system admins now? I ask, only because I know people have been coming and going recently. Frankly, for the size of the site, they do really, really well for the limited resources they have.

I meant admins in the more general purpose sense of administrators, people who are paid to maintain the system. But yeah I agree, quality:resources ratio is really really high.

Re: Amazon Web Services are down

#314

Post by scorpion032 » Thu, Apr 21, 2011, 6:46 PM UTC

ry0ohki wrote: post #256

BrandonM wrote: post #243

Earlier quoted context omitted.

Why does Reddit really need 99% availability? Is a customer unduly harmed or is the world even worse off if Reddit is down for a couple cumulative days per year? Is it worth the cost? Would you put up with more ads and/or pay for Reddit in order to make sure that it's available 24/7/365?

Probably not as much for the customer as for the company. When sites are unreliable, people end up going to the more reliable competitors as they arise.

If any site can take down time and not lose users, reddit can. And it has.

Re: Amazon Web Services are down

#315

Post by pextris » Thu, Apr 21, 2011, 6:48 PM UTC

reddit.com is down, but luckily http://radioreddit.com is not.

Re: Amazon Web Services are down

#316

Post by oomkiller » Thu, Apr 21, 2011, 6:53 PM UTC

jedberg wrote: post #238

bvi wrote: post #225

Earlier quoted context omitted.

You guys might have answered this in one of your AMAs/blog posts (or was it raldi who commented?), but what options can reddit resort to should this stuff happen again to this degree of severity?

We're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option. So I guess that is the long way of saying that hopefully it won't happen again.

Please tell us how you plan on moving 700 EBS volumes to something completely different. It sounds amazing.

Re: Amazon Web Services are down

#317

Post by scorpion032 » Thu, Apr 21, 2011, 6:53 PM UTC

tedjdziuba wrote: post #193

stef25 wrote: post #132

Earlier quoted context omitted.

Reddit's been down for several hours today, I'm sure they are already way lower 99.95%.

Reddit being down is not news.

You made Google App Engine managers cry once: http://www.theregister.co.uk/2009/07/06/dziuba_google_app_en...

Anything coming up for amazon? If not for anything else, for pure entertainment value!

Re: Amazon Web Services are down

#318

Post by brianpan » Thu, Apr 21, 2011, 6:56 PM UTC

dfranke wrote: post #293

campnic wrote: post #223

Earlier quoted context omitted.

"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.

I'm actually surprised if incurring 50% extra hardware costs really is cheaper than the cost of being down. If Netflix is down for a few hours, then it costs them some goodwill, and maybe a few new signups, but is the immediate revenue impact really that great? Most of Netflix's revenue comes from monthly subscriptions, and it's not like their customers have an SLA.

Obviously you haven't been around my wife when she loses the last 5 minutes of a show. SLA or no, services will get cancelled.

Re: Amazon Web Services are down

#319

Post by imajes » Thu, Apr 21, 2011, 7:15 PM UTC

dfranke wrote: post #293

campnic wrote: post #223

Earlier quoted context omitted.

"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.

I'm actually surprised if incurring 50% extra hardware costs really is cheaper than the cost of being down. If Netflix is down for a few hours, then it costs them some goodwill, and maybe a few new signups, but is the immediate revenue impact really that great? Most of Netflix's revenue comes from monthly subscriptions, and it's not like their customers have an SLA.

Actually, they do. and Netflix proactively refund customers for downtime. Usually it's pennies on the dollar, but i've had more than refund for sub 30 minute outages which have prohibited me from using the service.

Netflix are very very sensitive to this problem because it's much harder for them to sell against their biggest competitor (local cable) since they rely on the cable to deliver their service. If the service goes down, then the cable company can jump in and say, "You'll never lose the signal on our network" -- blatantly untrue, but it doesn't matter.

When you're disrupting a market, remember that what seem trivial is in fact hugely important when you're fighting huge well-established competition :)

Re: Amazon Web Services are down

#320

Post by ghotli » Thu, Apr 21, 2011, 7:19 PM UTC

pvh wrote: post #124

jedberg wrote: post #15

Earlier quoted context omitted.

Thankfully, no. :) But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.

Hey Jedberg, if you guys aren't already rolling your own, check out fdr's WAL-E tool. It bounces postgres write-ahead logs off S3 and goes great with the new PG9 replication. https://github.com/heroku/WAL-E

Thanks for this. I had designed and partially implemented this exact same thing. Do you know of this running in production anywhere?