Why did we take reddit down for 71 minutes?

#1

Post by icey » Thu, Jan 07, 2010, 8:44 PM UTC

Why did we take reddit down for 71 minutes?
blog.reddit.com

Re: Why did we take reddit down for 71 minutes?

#2

Post by jbellis » Thu, Jan 07, 2010, 9:28 PM UTC

"Memcachedb also has another feature that blocks all reads while it writes to the disk."

Seriously?

Wow.

Re: Why did we take reddit down for 71 minutes?

#3

Post by davidw » Thu, Jan 07, 2010, 9:32 PM UTC

jbellis wrote: post #2

"Memcachedb also has another feature that blocks all reads while it writes to the disk." Seriously? Wow.

Sounds like it's time to look at Redis...

Re: Why did we take reddit down for 71 minutes?

#4

Post by rubyrescue » Thu, Jan 07, 2010, 9:43 PM UTC

sounds like a redefinition of 'in any way' to me...

So why am I singing the praises of Amazon and EC2? Mainly to dispel the opinion that the site getting slower since the move is in any way related to Amazon...(snipped)...Unfortunately, the single EBS volumes they were on could not handle these bursting writes.

Re: Why did we take reddit down for 71 minutes?

#5

Post by simonw » Thu, Jan 07, 2010, 9:44 PM UTC

davidw wrote: post #3

jbellis wrote: post #2

"Memcachedb also has another feature that blocks all reads while it writes to the disk." Seriously? Wow.

Sounds like it's time to look at Redis...

Or Tokyo Cabinet / Tyrant, which is still a high performance key/value store but doesn't need to fit everything in RAM. Depends on how much they're storing.

Re: Why did we take reddit down for 71 minutes?

#6

Post by ShabbyDoo » Thu, Jan 07, 2010, 9:45 PM UTC

I'm bothered by the need to run SW RAID on top of HW RAID. One would think that Amazon would sell faster EBS "disks" for a premium.

And slower disks for a discount? But, I guess that's what S3 is for.

Re: Why did we take reddit down for 71 minutes?

#7

Post by psranga » Thu, Jan 07, 2010, 9:51 PM UTC

ShabbyDoo wrote: post #6

I'm bothered by the need to run SW RAID on top of HW RAID. One would think that Amazon would sell faster EBS "disks" for a premium. And slower disks for a discount? But, I guess that's what S3 is for.

I think it works like this:

Even if Amazon uses 'k' hdds for one EBS "disk", since you're sharing the real hdds with other users, you don't get 'k' hdds' performance, you only get a fraction.

By RAIDing over 'n' EBS "disks", you are effectively compensating for the reduced performance due to sharing.

Re: Why did we take reddit down for 71 minutes?

#8

Post by ShabbyDoo » Thu, Jan 07, 2010, 9:59 PM UTC

psranga wrote: post #7

ShabbyDoo wrote: post #6

I'm bothered by the need to run SW RAID on top of HW RAID. One would think that Amazon would sell faster EBS "disks" for a premium. And slower disks for a discount? But, I guess that's what S3 is for.

I think it works like this: Even if Amazon uses 'k' hdds for one EBS "disk", since you're sharing the real hdds with other users, you don't get 'k' hdds' performance, you only get a fraction. By RAIDing over 'n' EBS "disks", you are effectively compensating for the reduced performance due to sharing.

I get what the stack looks like, but it seems really broken and likely quite inefficient. Thus far, Amazon has gone after greenfield applications which can be written within the constraints of their cloud platform. However, there are a ton of people hosting their own SQL database-based apps where a single DB is the bottleneck. Without significant refactoring, these apps can only scale vertically with the DB. So, while Amazon provides nice, big boxes to run SQLServer/MySQL/etc., disk performance is that of a desktop machine -- hardly a balanced system. How many more customers could they capture if they offered premium, high-performance storage options?

Re: Why did we take reddit down for 71 minutes?

#9

Post by [deleted] » Thu, Jan 07, 2010, 10:02 PM UTC

[deleted]

Re: Why did we take reddit down for 71 minutes?

#10

Post by aaronblohowiak » Thu, Jan 07, 2010, 10:29 PM UTC

rubyrescue wrote: post #4

sounds like a redefinition of 'in any way' to me... So why am I singing the praises of Amazon and EC2? Mainly to dispel the opinion that the site getting slower since the move is in any way related to Amazon...(snipped)...Unfortunately, the single EBS volumes they were on could not handle these bursting writes.

This isn't an EC2 issue, this is a SAN issue. Wether it is EBS or an NFS drive, meh. This is an architecting issue and while it is a result of the underlying hardware, the underlying hardware is not the constraint.