How does RethinkDB perform when compared to open-source distributed databases built with hate?
Maybe not built with hate, but used in anger?
RethinkDB: An open-source distributed database built with love over three years
121–130 of 247 posts
Re: RethinkDB: An open-source distributed database built with love over three years
#122How do filters work? They seem pretty difficult implementation-wise since you can write them in any of the language bindings. My first guess is that you pipe all the data in a table to the client, and the client itself does the filtration. But this would be extraordinarily inefficient.
Piping all the data to the client would be extremely inefficient. Fortunately we don't do that. When a filter is written in the client language it gets compiled into a protocol buffer which is sent to the cluster. This gets compiled into a query which is sent to each of the relevant shards for the table. This query has the filter baked right into it. The shards then go through their local copy of the data and filter…
To add to jdoliner's answer, the reason why you can write table('foo').filter(lambda x: x['bar'] > 5).run() is because we do some language trickery on the client side to compile the query to an AST. In this case, we overload greater than operator, call the lambda function once on the client with a special object, and return an AST. This AST is then sent to the server and executed there.
It's rather difficult to integrate into a host language like that smoothly from the driver implementation perspective, but once the driver is written the user experience is amazing because you can write queries that look exactly like Python, but they're executed entirely on the server.
Re: RethinkDB: An open-source distributed database built with love over three years
#123Earlier quoted context omitted.
There is of course no fundamental reason why JSON-based db's has to be schemaless. This is one interesting direction that might be worth exploring.
I would love a system that is schema-less by design, but has guards that can be enforced at insert/update. That way, the underlying data structures don't have to be locked up from complex migrations (as needed w/ ALTER TABLE), but you still get type safety. A migration instead would simply involve a change in guards and an asynchronous update of existing entries. Plus you'd get all the wins of something resembling op…
This is a feature we've talked a lot about. Another idea we think is interesting is having the database detect schema such that users could see a readout that said:
100% of your documents have a integer field named "foo" would you like to make this a schema constraint?
Re: RethinkDB: An open-source distributed database built with love over three years
#124Wow, looks great! I'm excited to try it out.
The Github graphs are really interesting too, that's a lot of love/work right there!
Re: RethinkDB: An open-source distributed database built with love over three years
#125I'll ask the obvious question not in the FAQ: How is this different from MongoDB?
One apparent difference between RethinkDB and MongoDB is that in RethinkDB, you can only index on the primary key. I imagine secondary indexes will be coming along soon.
We're planning to add secondary indexes in the next few releases. Doing them well in distributed systems is really hard. It's relatively easy to check off a box and introduce it as a feature, to make it actually behave well and allow for good performance is incredibly tricky. This is why they didn't make it into this release -- since it's a core feature of every db, we wanted to take the time to do this right.
Re: RethinkDB: An open-source distributed database built with love over three years
#126Earlier quoted context omitted.
Hi, here to answer question number 4. Short answer: Our configuration data is most similar to git. Any machine can be used as an administrative node via the WebUI or the CLI. It will make changes to the metadata which then get pushed to the other nodes. If 2 nodes make conflicting changes you get a conflict which the system will help you to merge. Long Answer Cluster configuration is stored in semilattices which are…
Interesting. Do you have any way of checking that the change has actually propagated through the system before starting to act on it? Is the system consistent at all times? If I understand correctly, the client can connect to any instance and its request will get routed appropriately. Let's assume that you take a master offline and promote one of the replicas to be a new master. Won't that lead to a window in which (…
Joe may be responding to this soon, but in the meantime I'll chime in. There is no way to verify the propagation reliably without either introducing strong performance inefficiencies (e.g. two phase commit protocol), or divergence (paxos, semi lattices, etc.) In our implementation we're using immediately consistent algorithms for data, but eventually consistent algorithms for cluster metadata. This means that if there is a metadata conflict, the user is presented with an issue (via the web ui or CLI) that they have to resolve. We'll also be adding automated resolution soon.
We basically have something very similar to zookeper baked into rethinkdb. We wrote it internally from scratch to better suit the needs of our architecture.
Re: RethinkDB: An open-source distributed database built with love over three years
#127Are there plans for an Arc driver?
Re: RethinkDB: An open-source distributed database built with love over three years
#128The querying capabilities here look amazing. Having to manually figure out how do joins and group by in something like CouchDB is really a pain, but this looks really slick. Very impressed and I will be trying this out!
We spent a long time trying to reimplement other people's protocols but with our engine underneath. Being able to have features like this is one of the things that eventually convinced us it would be worth it to control our own. Bear in mind that our joins are also distributed which we think is really cool.
Re: RethinkDB: An open-source distributed database built with love over three years
#129What the heck does "built with love" even mean? Is this just a hipster marketing term to tell us that it's small and cute and made by people who play ukuleles and ride unicycles in their spare time, and not by evil corporate people who commute to work and have mortgages? I find a lot of advertising eyeroll inducing, and the current trend of more-hipster-than-thou posturing is right at the top.
What does hipster even mean here?
Re: RethinkDB: An open-source distributed database built with love over three years
#130Hey guys, Slava here. I've been up since yesterday, so I'm going to clock out (though some of the team members are still lurking here). I wanted to thank everyone for great feedback. We're working hard to improve Rethink over the next few months. FYI, you can always hop on IRC (#rethinkdb on freenode) or github tracker (https://github.com/rethinkdb/rethinkdb/issues) with questions and we'll help you out.