Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

infoq.com

Post reply on HN
Print view View original

101–110 of 114 posts

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#101

Post by dpatrick86 »

kerkeslager wrote: post #98
diminish wrote: post #76

Earlier quoted context omitted.

The dominance of non-relevance is interesting for HN. Was writing exact the same thing, and just saw yours. In this and some other technical topics, people end up discussing their personal tastes with web site's design, their individual UI frustrations with some button on the web site, the font, the color, and other random non-relevant topic; like now the profitability.

It's no excuse, but I see a reason for this: most people feel they understand these irrelevant topics better than they understand the scalability features of the infrastructure Reddit has built. People talk about their comfort zone.

Spot on observation IMO

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#102

Post by jedberg »

hardwaresofton wrote: post #83

So did anyone else take the most important lessons in the video as: 1. Use AWS 2. Use Postgres 3. Use AWS 4. Use Cassandra 5. Use python, so later you can write C when shit needs to go super fast That's what I got.

I find this interesting, this is really great feedback for me actually.

Those are some of the important lessons, although use (postgres|cassandra) are really too prescriptive. More like "use the right tool or tools for the job".

Also, use consistent key hashing where appropriate is another important less that I should emphasize more.

And "build for 3" is another important lesson. It makes scaling much easier.

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#103

Post by jedberg »

robotmay wrote: post #91
jedberg wrote: post #80

Earlier quoted context omitted.

Sorry that wasn't clear. The latency didn't get better, but what happened is that instead of having to make a lot of calls to memcache it was just one (well, just a few), so while that one took longer, the total time was much less.

I actually did some (simplistic) examples of this in a small presentation to illustrate the performance improvements of batching memcached requests, if anyone's interested: https://speakerdeck.com/robotmay/a-simple-introduction-to-ef... (slides 11 to 14)

That's a better explanation than mine. :) Thanks for the link.

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#104

Post by jedberg »

conradfr wrote: post #95

I wonder if the demise of Digg three years ago and the (supposedly) inflow of new users have been problematic at the time.

There wasn't actually a very large jump in traffic when Digg v4 was launched. Most of those folks were already reddit users. Traffic bumped a little bit, but not all that much.

It's important to keep in mind that reddit was already doing twice as much traffic as Digg before they launched v4.

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#105

Post by jedberg »

heed wrote: post #90

Side question: is vote queuing the reason behind the sometimes large drops in score on highly active and popular submissions on reddit?

No. That's generally because once something gets popular and jumps to the front page, it gets a huge boost in visibility, especially from people who weren't looking at the niche subreddit it comes from.

A lot of those people aren't interested in that content, so it will suddenly get an influx of downvotes.

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#106

Post by jedberg »

vegasje wrote: post #100

Does anyone know how the different storage systems are utilized, and why each system is utilized for that purpose? The presenter mentions using memcached, Cassandra, and PostgreSQL, and mentions the same type of data when discussing each (votes, for instance). I would definitely benefit from a more in-depth understanding of how each system is utilized, and why.

Each tool has a different use case. Votes is a great example.

Memcache has no guarantees about durability, but is very fast, so the vote data is stored there to make rendering of pages as quick as possible.

Cassandra is durable and fast, and gives fast negative lookups because of its bloom filter, so it was good for storing a durable copy of the votes for when the data isn't in memcache.

Postgres is rock solid and relational, so it was a good place to store votes as a backup for Cassandra (we could regenerate all the data in Cassandra from Postgres if necessary) and also for doing batch processing, which sometimes needed the relational capabilities.

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#107

Post by vegasje »

jedberg wrote: post #106
vegasje wrote: post #100

Does anyone know how the different storage systems are utilized, and why each system is utilized for that purpose? The presenter mentions using memcached, Cassandra, and PostgreSQL, and mentions the same type of data when discussing each (votes, for instance). I would definitely benefit from a more in-depth understanding of how each system is utilized, and why.

Each tool has a different use case. Votes is a great example. Memcache has no guarantees about durability, but is very fast, so the vote data is stored there to make rendering of pages as quick as possible. Cassandra is durable and fast, and gives fast negative lookups because of its bloom filter, so it was good for storing a durable copy of the votes for when the data isn't in memcache. Postgres is rock solid and re…

That makes a lot of sense. Were the majority of your systems using this "durability chain" so to speak -- memcached -> Cassandra -> Postgres? Additionally, in retrospect do you find this type of chain to work well, and would you use it again (perhaps you already are over at Netflix)?

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#108

Post by alanctgardner2 »

kerkeslager wrote: post #98
diminish wrote: post #76

Earlier quoted context omitted.

The dominance of non-relevance is interesting for HN. Was writing exact the same thing, and just saw yours. In this and some other technical topics, people end up discussing their personal tastes with web site's design, their individual UI frustrations with some button on the web site, the font, the color, and other random non-relevant topic; like now the profitability.

It's no excuse, but I see a reason for this: most people feel they understand these irrelevant topics better than they understand the scalability features of the infrastructure Reddit has built. People talk about their comfort zone.

This is the fundamental rule of bikeshedding, which HN has discussed, decried and regurgitated since time immemorial.

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#109

Post by conradfr »

jedberg wrote: post #104
conradfr wrote: post #95

I wonder if the demise of Digg three years ago and the (supposedly) inflow of new users have been problematic at the time.

There wasn't actually a very large jump in traffic when Digg v4 was launched. Most of those folks were already reddit users. Traffic bumped a little bit, but not all that much. It's important to keep in mind that reddit was already doing twice as much traffic as Digg before they launched v4.

Thanks for your input, didn't know that.

I, for one, went from a /r/php lurker to real user at that time so I thought there was dozens of us, dozens !

Re: Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]

#110

Post by heed »

jedberg wrote: post #105
heed wrote: post #90

Side question: is vote queuing the reason behind the sometimes large drops in score on highly active and popular submissions on reddit?

No. That's generally because once something gets popular and jumps to the front page, it gets a huge boost in visibility, especially from people who weren't looking at the niche subreddit it comes from. A lot of those people aren't interested in that content, so it will suddenly get an influx of downvotes.

Thanks for the reply.

I'm glad there is an explanation based on user behavior for this phenomenon because admin level vote tampering is such a tired theory.

Post reply on HN