Amazon Web Services are down • Hacker News Board

Re: Amazon Web Services are down

#331

Post by ulope » Thu, Apr 21, 2011, 9:50 PM UTC

acangiano wrote: post #170

stef25 wrote: post #127

Earlier quoted context omitted.

Amazon is really not being kind to you guys; I sort of hope you'll find an alternative solution fast!

If I was Rackspace, I'd be at Reddit/Wired's headquarters already.

Rackspace (and really all the "popular" US hosters) seem ridiculously expensive compared to hosting prices we have in Germany (see e.g. http://www.hetzner.de/en/hosting/produktmatrix/rootserver-pr... this is one of the biggest root server hosters in Germany).

Is this really so or are Racksoace and co. Just "boutique" offerings?

Re: Amazon Web Services are down

#332

Post by adrianco » Thu, Apr 21, 2011, 9:51 PM UTC

jedberg wrote: post #240

campnic wrote: post #223

Earlier quoted context omitted.

"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.

At their level of income, this is true. For us, we are just now staffing up to the level where we can make the changes necessary to do the same thing.

I think it's incredible that you guys can run a site at all with the few people you've got. Hope it all gets better again soon.

Re: Amazon Web Services are down

#333

Post by adrianco » Thu, Apr 21, 2011, 9:55 PM UTC

dfranke wrote: post #293

campnic wrote: post #223

Earlier quoted context omitted.

"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.

I'm actually surprised if incurring 50% extra hardware costs really is cheaper than the cost of being down. If Netflix is down for a few hours, then it costs them some goodwill, and maybe a few new signups, but is the immediate revenue impact really that great? Most of Netflix's revenue comes from monthly subscriptions, and it's not like their customers have an SLA.

There isn't a 50% extra hardware cost. You spread systems over three zones and run at the normal utilization levels of 30-60%. If you lose a zone while you are at 60% you will spike to 90% for a while, until you can deploy replacement systems in the remaining zones. Traffic spikes mean that you don't want to run more than 60% busy anyway.

Re: Amazon Web Services are down

#334

Post by pbhjpbhj » Thu, Apr 21, 2011, 10:05 PM UTC

jedberg wrote: post #249

pbhjpbhj wrote: post #230

Earlier quoted context omitted.

Didn't Jedburg say that they could reduce the failure by spending with Amazon. I wonder if Rackspace really want this particular traffic burden. It seems that if Reddit choose not to pay for the load they need then you get lot's of bad press for it ... perhaps I'm seeing it wrong. Rubbish analogy: Kinda like if I was doing a haulage business and you called out for a wheelbarrow to carry some elephants, then when the…

No, I said that we have spent all we can, and at this point we need development. However, in this case, the outage is not because of any issues with our setup, but with Amazon.

>"we have spent all we can"

So is it a financial constraint with Amazon? Would you be suffering the same sorts of outages regardless of the technology on the backend or does AWS basically suck?

Re: Amazon Web Services are down

#335

Post by mtodd » Thu, Apr 21, 2011, 10:37 PM UTC

mtodd wrote: post #283

Why is ELB not mentioned at all on the Service Health Dashboard? We're experiencing problems with two of our ELBs, one indicating instance health as out of service, reporting "a transient error occurred". Another, new LB (what we hoped would replace the first problematic LB), reports: "instance registration is still in progress". A support issue with Amazon indicated that it was related to the ongoing issues and to m…

Quote from the AWS support rep: "I can confirm that ELB has been affected by the EBS issue despite the lack of messaging on the AWS Dashboard".

Re: Amazon Web Services are down

#336

Post by [deleted] » Thu, Apr 21, 2011, 11:16 PM UTC

yuvadam wrote: post #2

Current status: bad things are happening in the North Virginia datacenter. EC2, EBS and RDS are all down on US-east-1. Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

[deleted]

Re: Amazon Web Services are down

#337

Post by freakball » Fri, Apr 22, 2011, 12:19 AM UTC

jedberg wrote: post #328

mattdeboard wrote: post #266

Earlier quoted context omitted.

I can't imagine how frustrating the jobs of the Reddit admins must be.

It's usually very rewarding. The awesome community is what keeps me doing it.

Awesome, I just figured out that you kept the votes tallied during the 'downtime'. What's interesting is how clearly good and bad submissions were dichotomized when nobody had anything else to vote on.

Re: Amazon Web Services are down

#338

Post by Devilboy » Fri, Apr 22, 2011, 12:25 AM UTC

obfuscate wrote: post #324

epi0Bauqu wrote: post #274

Earlier quoted context omitted.

How do you move away from EBS and still deal with large data?

Not sure what you had in mind by "large", but instance storage goes up to 1.7TB: http://aws.amazon.com/ec2/instance-types/

The reason Reddit uses RAID10 is for performance, not disk size. A single instance storage device is just too slow for the Reddit database.

Re: Amazon Web Services are down

#339

Post by shrike » Fri, Apr 22, 2011, 1:06 AM UTC

jedberg wrote: post #238

bvi wrote: post #225

Earlier quoted context omitted.

You guys might have answered this in one of your AMAs/blog posts (or was it raldi who commented?), but what options can reddit resort to should this stuff happen again to this degree of severity?

We're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option. So I guess that is the long way of saying that hopefully it won't happen again.

NOTE: I work for Gluster.

We have had a lot of success stabilizing EBS by creating mdadm arrays out of lots of smaller EBS volumes. There is minimal additional costs and you can get better performance, stability, and protection (RAID 5, 6).

Gluster makes an OSS distributed filesystem that runs across availability zones, our AMI (not OSS) builds multiple RAID arrays on each instance then spreads the filesystem across instances in multiple AZs. Send me an email if you want to chat.

Re: Amazon Web Services are down

#340

Post by redthrowaway » Fri, Apr 22, 2011, 5:03 AM UTC

jedberg wrote: post #240

campnic wrote: post #223

Earlier quoted context omitted.

"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.

At their level of income, this is true. For us, we are just now staffing up to the level where we can make the changes necessary to do the same thing.

Are you looking to diversify across ebs or set up dedicated hosting?