Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reddit: 2012 State of the Servers

blog.reddit.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: Reddit: 2012 State of the Servers

#51

Post by notaddicted »

fleitz wrote: post #32
x3c wrote: post #24

Earlier quoted context omitted.

But how do you achieve data persistence in case of server crash? Snapshots are not reliable for that, slave db servers aren't foolproof either.

You just copy the WAL log to another server and replay it. It takes a day to setup and test. Once that is setup you have two options async replication (which means you'll lose about 100ms of data in event of a crash) or you can use sync replication which means the transaction doesn't commit until the WAL log is replicated on the other server. (that adds latency but doesn't really affect throughput) I'm not exactly su…

Another possibility on the AWS cloud is to put the logs on S3, Amazon S3 has High Durability. Heroku has published a tool for doing it: https://github.com/heroku/WAL-E , which they use to manage their database product.

Re: Reddit: 2012 State of the Servers

#52

Post by techscruggs »

onemoreact wrote: post #50
rbranson wrote: post #33

Earlier quoted context omitted.

This is because people came to the table with unrealistic expectations. They were used to dealing with mature software based on decades old proven ideas and coming into very experimental territory expecting to get a smooth experience. Cassandra has enabled Reddit to manage a highly scalable distributed data store with a tiny staff. This is not to say it has been trouble free, but it has enabled them to do something t…

I respect the Reddit team, but I don't think they need to use Cassandra at their scale. I mean they only have 2TB of data in total. They should easily be able to use a simple caching system to keep the last 2 weeks of data in RAM and basically never read from the database. That said, they may be freaked out based on their growth curve and simply thinking ahead.

They said that they had 2TB in postgres not 2TB of total data. I imagine all of their data is probably about an order of magnitude larger. Additionally, the challenges are not as much around how much data you have, but how you want to access that data (indexes).

Re: Reddit: 2012 State of the Servers

#54

Post by rbranson »

techscruggs wrote: post #52
onemoreact wrote: post #50

Earlier quoted context omitted.

I respect the Reddit team, but I don't think they need to use Cassandra at their scale. I mean they only have 2TB of data in total. They should easily be able to use a simple caching system to keep the last 2 weeks of data in RAM and basically never read from the database. That said, they may be freaked out based on their growth curve and simply thinking ahead.

They said that they had 2TB in postgres not 2TB of total data. I imagine all of their data is probably about an order of magnitude larger. Additionally, the challenges are not as much around how much data you have, but how you want to access that data (indexes).

Indeed. It boils down to their need for a durable cache. It's simply too expensive to try to cache every comment tree in RAM, and Cassandra's data model and disk storage layout is a really good fit for the structure of their data.

Re: Reddit: 2012 State of the Servers

#55

Post by ruckusing »

x3c wrote: post #24
joevandyk wrote: post #13

They say they moved off ebs and onto local storage for postgres and saw a big increase in reliability and performance. I did the same for my site last year and it was great. This is one of the reasons why I haven't moved my Postgres databases to enterprisedb or heroku: they use ebs.

But how do you achieve data persistence in case of server crash? Snapshots are not reliable for that, slave db servers aren't foolproof either.

We have the same setup, use local ephemeral disks on EC2 with Postgres. We never even tried EBS as we just heard too much negative things about it, namely its variance in performance.

So our approach is to RAID-10 (4) local volumes together. We then use replication to at least 3 slaves, all of which are configured the same and can become master in the event of a failover.

We use WAL-E[0] to ship WAL logs to S3. WAL-E is totally awesome. Love it!

[0] https://github.com/heroku/wal-e

Re: Reddit: 2012 State of the Servers

#56

Post by onemoreact »

rbranson wrote: post #54
techscruggs wrote: post #52

Earlier quoted context omitted.

They said that they had 2TB in postgres not 2TB of total data. I imagine all of their data is probably about an order of magnitude larger. Additionally, the challenges are not as much around how much data you have, but how you want to access that data (indexes).

Indeed. It boils down to their need for a durable cache. It's simply too expensive to try to cache every comment tree in RAM, and Cassandra's data model and disk storage layout is a really good fit for the structure of their data.

You don't need every comment tree in RAM just the last few days worth plus a few older threads that get linked back to. They are currently using 200 machines so let's say 10 of them are used to cache 1 weeks comments. 30 GB of ram * 10 machines = 300GB of cache. I would be vary surprised if they generate 200GB/week or 10TB of comment data a year.

Edit: For comparison Slashdot spent a long time on just 6 less powerful machines vs the 200+ Reddit is using. Reddit may have more traffic, but not 40x as much. And, last I heard HN just uses one machine.

PS: The average comment is small and they can compress most comments after a day or so. They can probably get away with storing a second copy of most old threads as a blob of data in case people actually open it which cost a little space, but cuts down on processing time.

Re: Reddit: 2012 State of the Servers

#57

Post by nbm »

rplnt wrote: post #47
ypcx wrote: post #41

Wondering how much of that 2TB dataset is necessary for the common daily functionality of reddit, probably less than 1%, and the rest is historical data, accessed by almost no one, except perhaps by the submission-dupe- checking algorithms, and similar?

Are you suggesting moving that down the ladder and not having that data everywhere? I.e. if someone want to see an old post, there would be one extra step required to load the data (so cdn, cassandra, now subset of postgres with not so old data, "full" postgres). I think facebook does something similar, but they really have to, considering their size.

In terms of status updates (ie, stories which may mention check-ins or photos or similar, but not Facebook Messages), before Facebook's Timeline launch, there were multiple stores of data depending on age. With Timeline, all the different versions of data over all ages were put back together into a single (logical) store. More about that process at:

    https://www.facebook.com/notes/facebook-engineering/building-timeline-scaling-up-to-hold-your-life-story/10150468255628920

Re: Reddit: 2012 State of the Servers

#58

Post by rbranson »

onemoreact wrote: post #56
rbranson wrote: post #54

Earlier quoted context omitted.

Indeed. It boils down to their need for a durable cache. It's simply too expensive to try to cache every comment tree in RAM, and Cassandra's data model and disk storage layout is a really good fit for the structure of their data.

You don't need every comment tree in RAM just the last few days worth plus a few older threads that get linked back to. They are currently using 200 machines so let's say 10 of them are used to cache 1 weeks comments. 30 GB of ram * 10 machines = 300GB of cache. I would be vary surprised if they generate 200GB/week or 10TB of comment data a year. Edit: For comparison Slashdot spent a long time on just 6 less powerful…

Please. Reddit does 2 billion page views per month.

Re: Reddit: 2012 State of the Servers

#59

Post by onemoreact »

rbranson wrote: post #58
onemoreact wrote: post #56

Earlier quoted context omitted.

You don't need every comment tree in RAM just the last few days worth plus a few older threads that get linked back to. They are currently using 200 machines so let's say 10 of them are used to cache 1 weeks comments. 30 GB of ram * 10 machines = 300GB of cache. I would be vary surprised if they generate 200GB/week or 10TB of comment data a year. Edit: For comparison Slashdot spent a long time on just 6 less powerful…

Please. Reddit does 2 billion page views per month.

Yea, and Casandra was built for a company serving 500 times that.

Re: Reddit: 2012 State of the Servers

#60

Post by ketralnis »

bru wrote: post #14
someone13 wrote: post #7

Earlier quoted context omitted.

Where do you get this estimate from? (Not disbelieving you, just curious)

One year and half ago, it was calculated and then confirmed by an admin[1] that the monthly cost was around 22K/month, or 270K/year. jedberg added that they were projecting to be around 350K/year by the end of 2010. Supposing that the cost increased linearly with the number of users (which sounds like a bad hypothesis, but is a start), the cost at the end of 2011 could be around 1M/year... That's impressive, but nowh…

A year and a half is a long time in reddit time
Post reply on HN