Live data from Hacker News

RethinkDB: An open-source distributed database built with love over three years

rethinkdb.com

71–80 of 247 posts

Re: RethinkDB: An open-source distributed database built with love over three years

#71
post #68

I'd really like to know why they don't have a PHP library, considering it powers half the web. The key shouldn't be to promote someone elses own language when using infrastructure products but rather to support everyone in using their tool. Mongo supports everything under the sun, so should this.

... and it will.

We had to start from somewhere and we also wanted to get a feel of what are the most requested libraries so we can focus our energy in those directions.

alex @ rethinkdb

Re: RethinkDB: An open-source distributed database built with love over three years

#72

I don't know much about RethinkDB yet, but I will say that I have been a big fan (online) of one of its founders, Slava Akhmechet, for years. I've never met him, but he wrote some terrific articles on his website, http://www.defmacro.org/ , a few years ago. Start at the bottom of the list of articles, with "The Nature of Lisp." Slava is a deep thinker, which makes me very excited to take a look at RethinkDB.

Indeed - he mentions in the article that he set himself a goal to convert 10 programmers into Lispers. Sounds like he probably has that many just in this thread! Kudos, sir!

Re: RethinkDB: An open-source distributed database built with love over three years

#74

Earlier quoted context omitted.

1. Yes -- that was the main motivation for MVCC. We wanted to allow people to use rethinkdb for analytics and map/reduce on top of the realtime system without dealing with having to replicate data into something else. 2. Short answer: we favor consistency (via master/slave under the hood). It allows for much easier API, much fewer issues in production, etc. The user experience is just better. If you're ok with out of…

I want to preface my comment: this is impressive work, congratulations on shipping, and this is what MongoDB should have been from the start. In reality, most transactional database deployments are heavily skewed towards read workload, so reading from hot slaves is basically a requirement for master/slave databases. So, in most real world applications at scale, apps already deal with inconsistencies between slaves an…

I mostly agree with what you're saying, but I also think there's enormous value in making easy things be really easy. Even with today's state of the art adding a shard, dealing with consistency issues, adding replicas, etc. is relatively hard. Perhaps not in a computer-sciency sense (all the problems are fairly well understood), but in an operational sense. Lots and lots of work needs to be done even with systems like MongoDB, let alone with MySQL. And once you're done with that, you can't really run complicated queries easily, so you have to solve that problem.

We set out to make these things be really easy (whether we succeed or not remains to be seen). We want the users not to have to deal with these issues at all whenever possible. You should be able to set up a cluster, add shards, and run cross-shard joins and aggregation in five minutes.

Of course once that problem is solved, there are tougher problems like high-performance cross-document distributed ACID, but I think the industry as a whole is relatively far away from that right now. (there are some solutions to this - e.g. Clustrix, but they require specialized hardware which makes it out of reach for most developers)

Re: RethinkDB: An open-source distributed database built with love over three years

#75

Please stop using json as a data model. I have no idea how to represent dates, or timestamps, or colors, or any other unsupported data type.

I have nothing to do with RethinkDB, but what are you talking about? Just represent them as strings. (What databases support colors as native types, anyways?) If you format dates YYYY-MM-DD, then you can do string comparisons for ranges.

And JSON has the huge advantage of supporting hierarchical data -- arrays with objects inside, etc. It seems a like a huge step forward.

Re: RethinkDB: An open-source distributed database built with love over three years

#76
post #69

Earlier quoted context omitted.

Thanks -- will do in the next few days.

What's the elevator pitch? Maybe we can help you with those advantages if you can tell us right now.

The elevator pitch is: "Mongo's ease of use without the gotchas." We have a nice simple to use query language and quick setup process. But things like analytic queries like map reduce don't lock up the entire database. Our product aims to not be a ticking time bomb of technical debt.

Re: RethinkDB: An open-source distributed database built with love over three years

#77
post #67

Nice work! It seems that you are well aware of the tradeoffs that you are taking and communicating it openly in your documentation (and your choices seem to be very reasonable). I really like the tone of your communication – it seems essentially BS/koolaid free. 1. How much data can you put in one instance before seeing performance degradation? I know that you still working on good benchmarks – but do you have any ba…

1. As long as active dataset everything fits in RAM, performance will be great. E.g. you can have terabytes of data but as long as the actively accessed dataset is 2. We do do block-level replication. On each node of the btree we store replication timestamps. When a node asks for new data, we can cull away parts of the tree the node has almost instantly. So replication is very very efficient for most OLTP workloads. We don't have statement-level replication yet, so if you do a range update on a large table, we'll have to replicate data block by block. It'll take a while to add statement-based replication - we'd have to do a pretty significant refactoring to make it happen.

3. Either. Replicas are great for failover -- if the master dies, you just failover and a replica picks up where the master left off. If you're ok with out-of-date reads, you can also hit replicas directly (e.g. for reports, etc.) and spread out the read load across the cluster.

4. This is a really complex question - we didn't document this because doing it properly would take a lot of time. I'll ask jdoliner to chime in -- he designed the architecture and wrote most of the code, perhaps he can describe it succinctly while we write deeper docs on this :)

5. We use protocol buffers between the client drivers and the server. We picked that because there were libraries for the initial three languages we picked (Ruby, JS, Python), they were really easy to use, and very efficient. We could also have a single spec for the client/server API. Internally we use our own serialization scheme which allows us to dump arbitrary C++ objects on the network. It doesn't support other languages (which we didn't need), but is much more versatile for writing complex cross-machine code.

Post reply on HN