Wondering how much of that 2TB dataset is necessary for the common daily functionality of reddit, probably less than 1%, and the rest is historical data, accessed by almost no one, except perhaps by the submission-dupe- checking algorithms, and similar?
Reddit: 2012 State of the Servers
41–50 of 62 posts
Re: Reddit: 2012 State of the Servers
#42Those are staggering numbers, glad i invested my time in reddit last year. We must be cautious of overheating though, signs of a bubble or a possible subreddit crisis.
Re: Reddit: 2012 State of the Servers
#43Could we get a public backup of the database already? Make it a torrent if bandwidth is an issue, but lets back that amazing resource up.
Re: Reddit: 2012 State of the Servers
#44Does anyone know what kind of hardware those 240 servers have? I wonder how much they cost.
They're EC2 large and x-large instance.
Re: Reddit: 2012 State of the Servers
#45It's interesting to see that they're sticking with Cassandra, and that they're having a much better experience with 0.8. I've been hearing so many fellow coders in SF hate on Cassandra that I had stopped considering it for projects. Has anybody worked with 0.8 or 1.0? Would you recommend Cassandra? I got to work with Riak a lot while I was at DotCloud, but the speed issue was pretty frustrating (it can be painfully s…
That's what we hear from our customers as well. They complain about excessive CPU and memory usage. The two phases we've seen are: 1/It's flexible and it works! Problem solved! 2/21st century called, they want their performance back. The problem with phase 2 is that you may not be able to solve it by throwing more computing power at it. Unfortunately if you really need map-reduce, at the moment I don't know what to r…
Be careful to compare apples to apples. Sure, the memory-only crowd (e.g, redis) will post higher numbers, but Cassandra is the performance leader for scalable, larger-than-memory datasets. See http://www.cubrid.org/blog/dev-platform/nosql-benchmarking/ for example. (And this tests an old version of Cassandra; we did a lot of optimization on the read path for 1.0: http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-...)
Re: Reddit: 2012 State of the Servers
#46Earlier quoted context omitted.
That's what we hear from our customers as well. They complain about excessive CPU and memory usage. The two phases we've seen are: 1/It's flexible and it works! Problem solved! 2/21st century called, they want their performance back. The problem with phase 2 is that you may not be able to solve it by throwing more computing power at it. Unfortunately if you really need map-reduce, at the moment I don't know what to r…
Be careful to compare apples to apples. Sure, the memory-only crowd (e.g, redis) will post higher numbers, but Cassandra is the performance leader for scalable, larger-than-memory datasets. See http://www.cubrid.org/blog/dev-platform/nosql-benchmarking/ for example. (And this tests an old version of Cassandra; we did a lot of optimization on the read path for 1.0: http://www.datastax.com/dev/blog/whats-new-in-cassand…
Didn't want to sound like someone scorning other people's work, I'm sure you did a lot of great things for 1.0.
However I have the gut feeling we're far from squeezing out all the juice from today's hardware.
Re: Reddit: 2012 State of the Servers
#47Wondering how much of that 2TB dataset is necessary for the common daily functionality of reddit, probably less than 1%, and the rest is historical data, accessed by almost no one, except perhaps by the submission-dupe- checking algorithms, and similar?
Are you suggesting moving that down the ladder and not having that data everywhere? I.e. if someone want to see an old post, there would be one extra step required to load the data (so cdn, cassandra, now subset of postgres with not so old data, "full" postgres). I think facebook does something similar, but they really have to, considering their size.
Re: Reddit: 2012 State of the Servers
#48Earlier quoted context omitted.
But how do you achieve data persistence in case of server crash? Snapshots are not reliable for that, slave db servers aren't foolproof either.
You just copy the WAL log to another server and replay it. It takes a day to setup and test. Once that is setup you have two options async replication (which means you'll lose about 100ms of data in event of a crash) or you can use sync replication which means the transaction doesn't commit until the WAL log is replicated on the other server. (that adds latency but doesn't really affect throughput) I'm not exactly su…
If this was for Postgres refer this :
http://wiki.postgresql.org/wiki/Binary_Replication_Tutorial
See the section "Starting Replication with only a Quick Master Restart", I tested this with a DB <10 GB. The setup time for replication was in the order of minutes. On a related note, it would be awesome if someone could point out real-world experiences for Postgres 9.1 synchronous replication.
Re: Reddit: 2012 State of the Servers
#49Re: Reddit: 2012 State of the Servers
#50It's interesting to see that they're sticking with Cassandra, and that they're having a much better experience with 0.8. I've been hearing so many fellow coders in SF hate on Cassandra that I had stopped considering it for projects. Has anybody worked with 0.8 or 1.0? Would you recommend Cassandra? I got to work with Riak a lot while I was at DotCloud, but the speed issue was pretty frustrating (it can be painfully s…
This is because people came to the table with unrealistic expectations. They were used to dealing with mature software based on decades old proven ideas and coming into very experimental territory expecting to get a smooth experience. Cassandra has enabled Reddit to manage a highly scalable distributed data store with a tiny staff. This is not to say it has been trouble free, but it has enabled them to do something t…
I respect the Reddit team, but I don't think they need to use Cassandra at their scale. I mean they only have 2TB of data in total. They should easily be able to use a simple caching system to keep the last 2 weeks of data in RAM and basically never read from the database.
That said, they may be freaked out based on their growth curve and simply thinking ahead.