Reddit explains their architecture, scaling and recent downtime
1–10 of 17 posts
Re: Reddit explains their architecture, scaling and recent downtime
#2I love how transparent reddit is about these sorts of things. In their video from pycon, they even discussed their server hosting bills!
Crazy how such a minor design decision (not using consistent hashing) could have such huge consequences down the line...
Re: Reddit explains their architecture, scaling and recent downtime
#3"It turns out that this one little decision makes it so that we can't horizontally scale that layer of our architecture without losing all the data already there (because all of the keys would point to the wrong server if we added a new one)."
I'm surprised they're looking to switch stacks entirely as opposed to a consistent hashing / distributed hash table (a la chord, dynamo, etc.)
Re: Reddit explains their architecture, scaling and recent downtime
#4"[memcached] can no longer return data fast enough for our needs, due to the way it interacts with BDb (its underlying data store)."
Does anyone know what the specific issue being alluded to here is?
Re: Reddit explains their architecture, scaling and recent downtime
#5"[memcached] can no longer return data fast enough for our needs, due to the way it interacts with BDb (its underlying data store)." Does anyone know what the specific issue being alluded to here is?
They're talking about memcachedb here rather than memcached.
It's memcache but with berkeleydb underneath for persistence.
Re: Reddit explains their architecture, scaling and recent downtime
#6"It turns out that this one little decision makes it so that we can't horizontally scale that layer of our architecture without losing all the data already there (because all of the keys would point to the wrong server if we added a new one)." I'm surprised they're looking to switch stacks entirely as opposed to a consistent hashing / distributed hash table (a la chord, dynamo, etc.) http://en.wikipedia.org/wiki/Cons…
We can't just switch hashing at this point, because none of the data would be in the right place.
Since we have to move all the data anyway, we figured now would be a good time to switch stacks. Memcachedb isn't really a very solid product, so even if we scale it, it isn't a good long term solution.
Re: Reddit explains their architecture, scaling and recent downtime
#7"[memcached] can no longer return data fast enough for our needs, due to the way it interacts with BDb (its underlying data store)." Does anyone know what the specific issue being alluded to here is?
MemcacheDB periodically flushes its in-memory DB to disk, using BDb. That causes a global read/write lock for a looooong time. (This pauses BCC for 5 seconds at a time when it happens, and I only have 20 MB stored in memcachedb. To rectify this I'm moving to Redis as soon as I get a day free. Suffice it to say that if Bingo Card Creator taxes the architecture of your key/value store, you may not quite be ready for prime time.)
Re: Reddit explains their architecture, scaling and recent downtime
#8"[memcached] can no longer return data fast enough for our needs, due to the way it interacts with BDb (its underlying data store)." Does anyone know what the specific issue being alluded to here is?
Yeah, that was a typo. Memcachedb just isn't fast enough. It used blocking IO when reading the disk, so if it is waiting on the disk for some data, none of the other requests can go through.
Re: Reddit explains their architecture, scaling and recent downtime
#9Two things surprise me about this article - probably because I've misunderstood it and don't see the big picture.
One is that there are master and slave databases and searches are done off the master - I've always seen them done off the slaves in other systems. The other is that they state that using MD5 doesn't allow for horizontal scaling. One of the qualities of MD5 is that all bits have an equal probability of being 0/1. Surely the last 1 or 2 bits can be used to indicate which server is holding the data?
Re: Reddit explains their architecture, scaling and recent downtime
#10"It is a highly customized user experience, on par with something like Facebook (just not as many users)"
Really, Reddit?
While, I don't doubt that they have complexities to deal with, this sounds like skewed perspective of either overestimating their own complexity, or underestimating Facebook's.