Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why did we take reddit down for 71 minutes?

blog.reddit.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Why did we take reddit down for 71 minutes?

#41

Post by antirez »

mrduncan wrote: post #23
simonw wrote: post #5

Earlier quoted context omitted.

Or Tokyo Cabinet / Tyrant, which is still a high performance key/value store but doesn't need to fit everything in RAM. Depends on how much they're storing.

Virtual memory support is currently being added to Redis so that it won't need to fit everything in RAM either. I'm sure antirez can provide some better info on how it'll work.

Hello, VM is already in alpha on Git actually. There is some more work to do, but I don't think in the Reddit use case there is the need of Redis VM: they are using MemcacheDB as a persistent cache, if it's a cache it should match very high performances delivered by memcached and is not required to cache everything. Redis is as fast or faster than memcached (using clients with the same performances) and is persistent, so probably it's a good fit for this problem.

Instead of VM Reddit should use Redis EXPIRE I guess, that is, time to live in cached keys so they auto expire.

Btw for people that don't know what Redis Virtual Memory is: with VM Redis is able to swap out rarely used keys in disk. This makes a lot of sense when using Redis as a DB. When using Redis as a cache, the way to go is EXPIRE: rarely used things in cache should simply go away and be expired instead of being moved into disk.

EDIT: It would be very interesting to know where the Reddit performance problem is, but Redis Sorted Sets are a very good match to create social-news alike sites like HN or Reddit in a scalable distributed way, with a few workers processing recent news to update their score into the sorted set. The home page can be generated with ZREVRANGE without any computation.

It's a shame reddit is not sharing how this cache is used.

Re: Why did we take reddit down for 71 minutes?

#42

Post by antirez »

jbellis wrote: post #32
davidw wrote: post #3

Earlier quoted context omitted.

Sounds like it's time to look at Redis...

Well, no; if they're using memcachedb presumably their data doesn't fit in memory on a single machine. Cassandra would be a better choice. /cassandra committer, but it would :)

When Redis is used as a persistent cache, the clients implementing consistent hashing are perfect fit for a distributed-redis, as you don't need all this data safeness. You can lose a node if there is a disaster without too much problems usually (like it happens in memcached), but you want, in normal conditions, that the cache is not volatile.

Re: Why did we take reddit down for 71 minutes?

#43

Post by jbellis »

jedberg wrote: post #35
jbellis wrote: post #32

Earlier quoted context omitted.

Well, no; if they're using memcachedb presumably their data doesn't fit in memory on a single machine. Cassandra would be a better choice. /cassandra committer, but it would :)

We actually looked at Cassandra and found it to be slower than memcachedb. However, we readily admit that we probably configured it wrong.

Hard to say, but we increased speed pretty much across the board about 50% from 0.4 to 0.4.2, by 50% (compounded :) to 0.5, and looking at 100% already for release-after-0.5... and we're ready to help configure on IRC :)

(Also, I'm not sure when you were looking at it, but bootstrap -- adding nodes without any downtime -- is done now.)

Re: Why did we take reddit down for 71 minutes?

#44

Post by jbellis »

antirez wrote: post #42
jbellis wrote: post #32

Earlier quoted context omitted.

Well, no; if they're using memcachedb presumably their data doesn't fit in memory on a single machine. Cassandra would be a better choice. /cassandra committer, but it would :)

When Redis is used as a persistent cache, the clients implementing consistent hashing are perfect fit for a distributed-redis, as you don't need all this data safeness. You can lose a node if there is a disaster without too much problems usually (like it happens in memcached), but you want, in normal conditions, that the cache is not volatile.

It sounded like they want to use it as more of a "real" database than a cache, since rebuilding data in case of a hardware failure is so painful.

Re: Why did we take reddit down for 71 minutes?

#45

Post by jedberg »

rms wrote: post #40
jedberg wrote: post #38

Earlier quoted context omitted.

That's not entirely true. Yes, it is true that getting servers was hard, and that was definitely a factor. But the bigger factor for me was that I was tired of having to build, image and rack all those servers. I liked the flexibility of EC2, and also not having to waste resources ordering a full rack's worth of hardware every time. Cost was also an issue. Datacenter space in SF is expensive, but it had to be in SF,…

OK, thanks for the clarification. I'm trying to find the post I read that gave me that idea but can't find it. Was it in in your AMA?

Could have been. Or possibly something Spez said.

Re: Why did we take reddit down for 71 minutes?

#46

Post by ericd »

nethergoat wrote: post #25
ericd wrote: post #13

Earlier quoted context omitted.

You hit the nail on the head as to why I'm looking into a physical DB server with RAIDed SSD's instead of hopping onto EC2. I would love to use Amazon and not have to deal with the potential headaches of managing physical machines, but the stories (maybe FUD) of having to raid EBS instances, spool up 20 instances to find the winners and kill the rest, etc etc really kills the appeal. If they could promise me consiste…

As someone who has spent the last year and a half running a 200-(persistent)node environment on EC2, including multiple m1.large and m1.xlarge DB pools, I can assure you those stories stem from FUD and unreasonable expectations. Yes, EBS is not very fast, especially compared to local disk. You can work around this, however, by configuring multiple volumes in a RAID configuration (as you have mentioned), or by scaling…

Thanks very much for the FUD-debunk, it's always great to get advice from someone who has thoroughly kicked the tires of something. I may start considering it once again.

Would you mind sharing what kind of small-block IO/sec numbers you've seen from the EBS's? My app tends to generate lots of IO with not a huge amount of cacheability, and it has a relatively small dataset, which is why I'm considering SSDs in the first place.

Re: Why did we take reddit down for 71 minutes?

#47

Post by rms »

jedberg wrote: post #45
rms wrote: post #40

Earlier quoted context omitted.

OK, thanks for the clarification. I'm trying to find the post I read that gave me that idea but can't find it. Was it in in your AMA?

Could have been. Or possibly something Spez said.

I thought it could have been a spez or kn0thing post but I went through all of them and couldn't find anything. I guess it could have been deleted, but I'm going to chalk this up to faulty memory on my part; you were there.
Post reply on HN