Amazon Web Services are down • Hacker News Board

Re: Amazon Web Services are down

#321

Post by joshhart » Thu, Apr 21, 2011, 7:42 PM UTC

jjm wrote: post #305

joshhart wrote: post #296

Earlier quoted context omitted.

I imagine they'd do consider some combination of the following (sorted by most likely) 1. Sharding data 2. Pulling tables out to other servers from the main DB 3. Pruning excessive data 4. Compressing data

It still has to be stored somewhere though right? If it's EBS you've just made yourself a complicated solution that will eventually fail all over again. No?

If the data is sharded, then the data/server is small enough enough to fit within the individual server's disk and you no longer need EBS to store it.

Re: Amazon Web Services are down

#322

Post by joshhart » Thu, Apr 21, 2011, 7:44 PM UTC

jjm wrote: post #303

jedberg wrote: post #238

Earlier quoted context omitted.

We're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option. So I guess that is the long way of saying that hopefully it won't happen again.

I do not believe you could be effective by moving away from EBS, you know without giving up quite a bit. Doing things the right way with EC2 means using EBS. It's the brake caliper to the rotor. Sure you could have drum brakes but they're not nearly anywhere effective as they quickly get heat soaked. I'm referring to S3. One should trust ephemeral storage. Your instance can go down at anytime. Write speeds to S3 are…

I can't say whether much has changed within the last year, but when I worked at FathomDB we had serious issues with EBS. You couldn't trust it. Odd things would happen like disks getting stuck in a reattaching state for days and disks having poor performance.

Re: Amazon Web Services are down

#323

Post by obfuscate » Thu, Apr 21, 2011, 8:14 PM UTC

wmoxam wrote: post #32

yuvadam wrote: post #2

Current status: bad things are happening in the North Virginia datacenter. EC2, EBS and RDS are all down on US-east-1. Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

Not all EC2 & EBS instances are down. I have several in US-east-1a and 1 is down, while all of the others are working.

I think which physical data center "us-east-1a" etc. corresponds to differs from user to user, to load-balance given that people will probably be more likely to use 1a than the other zones.

Re: Amazon Web Services are down

#324

Post by obfuscate » Thu, Apr 21, 2011, 8:15 PM UTC

epi0Bauqu wrote: post #274

jedberg wrote: post #238

Earlier quoted context omitted.

We're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option. So I guess that is the long way of saying that hopefully it won't happen again.

How do you move away from EBS and still deal with large data?

Not sure what you had in mind by "large", but instance storage goes up to 1.7TB: http://aws.amazon.com/ec2/instance-types/

Re: Amazon Web Services are down

#325

Post by Goladus » Thu, Apr 21, 2011, 8:17 PM UTC

jedberg wrote: post #240

campnic wrote: post #223

Earlier quoted context omitted.

"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.

At their level of income, this is true. For us, we are just now staffing up to the level where we can make the changes necessary to do the same thing.

I would also be shocked if Amazon isn't giving Netflix preferred pricing because it's such a high-profile customer.

Re: Amazon Web Services are down

#326

Post by obfuscate » Thu, Apr 21, 2011, 8:22 PM UTC

hoop wrote: post #299

bartonfink wrote: post #231

Earlier quoted context omitted.

Why wouldn't a simple expected value calculation work? You've shown that you can calculate the extra availability that subscribing to another region provides. Simply multiply the cost of an outage by the extra availability provided by an additional region that would have prevented that outage. If expanding to another region costs more than just taking the outage, then it's categorically not a good option. If manageme…

I think you're right, that would be the best way of presenting this argument to management. To do so, however, the company would need to calculate its Total Cost of Downtime (which probably isn't very complex for many companies) which is its own subject entirely IMO.

> calculate its Total Cost of Downtime (which probably isn't very complex for many companies)

Not complex even factoring in reputational damage?

Re: Amazon Web Services are down

#327

Post by mindcrime » Thu, Apr 21, 2011, 8:28 PM UTC

crb wrote: post #224

crb wrote: post #131

Earlier quoted context omitted.

Do we all regret letting GLaDOS reboot yet?

Sigh, Portal jokes were so much more popular on Reddit. :)

HN really isn't the place for internet memes, jokes about pop culture, and things that are judged trivial / frivolous. Part of what makes the HN community what it is, is a focus on high-quality, reasoned, rational discourse. IOW: HN != Reddit

Re: Amazon Web Services are down

#328

Post by jedberg » Thu, Apr 21, 2011, 8:29 PM UTC

mattdeboard wrote: post #266

jedberg wrote: post #244

Earlier quoted context omitted.

That hurts. But you're right, we've had a lot of issues. I think the reason this is news is because it is a massive Amazon failure.

I can't imagine how frustrating the jobs of the Reddit admins must be.

It's usually very rewarding. The awesome community is what keeps me doing it.

Re: Amazon Web Services are down

#329

Post by endgame » Thu, Apr 21, 2011, 9:41 PM UTC

spydez wrote: post #309

endgame wrote: post #164

Earlier quoted context omitted.

Interesting. Thanks. What did the Catholic Church have to say about that? Is it that Easter Monday is still on Monday, but the holiday is on Tuesday?

Easter is the holiday, and it's on Sunday. I'm pretty sure the Pope doesn't care much what people do the day after Easter (or the day after that).

Ah, thanks. I misread the wikipage, and didn't realise.

Re: Amazon Web Services are down

#330

Post by adrianco » Thu, Apr 21, 2011, 9:49 PM UTC

Goladus wrote: post #325

jedberg wrote: post #240

Earlier quoted context omitted.

At their level of income, this is true. For us, we are just now staffing up to the level where we can make the changes necessary to do the same thing.

I would also be shocked if Amazon isn't giving Netflix preferred pricing because it's such a high-profile customer.

Netflix pays standard rates for instances but uses reserved instances to pay less on bulk EC2 deployments