Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB: An open-source distributed database built with love over three years

rethinkdb.com

Post reply on HN
Print view View original

141–150 of 247 posts

Re: RethinkDB: An open-source distributed database built with love over three years

#141

Post by alexpopescu »

erichocean wrote: post #133
NathanKP wrote: post #99

Earlier quoted context omitted.

No according the website it is written in C++.

And yet the Clojure guy did his distributed DB in Clojure (aka, Lisp). Kind of makes me wonder why C++ was chosen...

I've joined RethinkDB just a couple of months ago, so I might not have all the historical facts right, but here is what I know.

In a previous incarnation RethinkDB was a highly optimized storage engine for SSDs implemented in C++ to be able to take full advantage of both low level SSD and kernel access.

The current distributed engine was built on top on this storage engine and I think it only made sense to continue with C++.

Re: RethinkDB: An open-source distributed database built with love over three years

#142

Post by oh_sigh »

erichocean wrote: post #134

I find JSON-oriented databases to be a huge limitation for writing applications managing any kind of financial data, due to the lack of a decimal number type and a timestamp/date type, both of which SQL provides (and are used A LOT). Sure, you can put that stuff in strings, but then you'll run into limitation with queries where you want to, e.g., aggregate a total, or do timestamp arithmetic. I could do everything wi…

Just store it in a string. Or a special map structure: {"date": "2012-11-01", "format":"YYYY-MM-DD" }

Re: RethinkDB: An open-source distributed database built with love over three years

#143

Post by spf13 »

jdoliner wrote: post #38
fooyc wrote: post #30

Earlier quoted context omitted.

> All queries are fully parallelized Does it means that every query touches all servers ? Or does it sends queries to only a subset of servers when possible ? (e.g. range queries on PK)

Joe Doliner - RethinkDB engineer here. > Does it means that every query touches all servers ? No. > Or does it sends queries to only a subset of servers when possible ? (e.g. range queries on PK) The query planner distributes the query between the nodes that actually contain the relevant data. Here are a few examples: In your example, a range get on the primary key, the query would touch one copy of each shard of the…

This is exactly how normal, map reduce and aggregation queries work in a sharded MongoDB cluster.

While it's true that on a single node MongoDB map reduce is single threaded, it is parallelized when running on a sharded cluster.

Re: RethinkDB: An open-source distributed database built with love over three years

#144

Post by oh_sigh »

ketralnis wrote: post #129
tjic wrote: post #39

What the heck does "built with love" even mean? Is this just a hipster marketing term to tell us that it's small and cute and made by people who play ukuleles and ride unicycles in their spare time, and not by evil corporate people who commute to work and have mortgages? I find a lot of advertising eyeroll inducing, and the current trend of more-hipster-than-thou posturing is right at the top.

What does hipster even mean here?

"People who I don't like and people who I think they think they're cooler than I am"

Re: RethinkDB: An open-source distributed database built with love over three years

#145

Post by alexpopescu »

erichocean wrote: post #134

I find JSON-oriented databases to be a huge limitation for writing applications managing any kind of financial data, due to the lack of a decimal number type and a timestamp/date type, both of which SQL provides (and are used A LOT). Sure, you can put that stuff in strings, but then you'll run into limitation with queries where you want to, e.g., aggregate a total, or do timestamp arithmetic. I could do everything wi…

We thought of supporting data types that are not part of JSON (date/time/timestamp/deltas, etc.), but we wanted to take the time to do it right so these didn't make it to this version.

alex @ rethinkdb

Re: RethinkDB: An open-source distributed database built with love over three years

#146

Post by coffeemug »

StavrosK wrote: post #138
coffeemug wrote: post #31

Earlier quoted context omitted.

1. Yes -- that was the main motivation for MVCC. We wanted to allow people to use rethinkdb for analytics and map/reduce on top of the realtime system without dealing with having to replicate data into something else. 2. Short answer: we favor consistency (via master/slave under the hood). It allows for much easier API, much fewer issues in production, etc. The user experience is just better. If you're ok with out of…

By the way, how safe is the JS interpreter? Can you get into trouble by running untrusted code in map/reduce queries?

JS interpreter (V8 under the hood) runs in a process pool -- similar to a thread pool, but outside of the core rethinkdb process. The code running in the JS interpreter cannot corrupt memory or crash the rethinkdb process (if it crashes, rethinkdb will simply start another v8 process). You also can't write from js executed on the server, so the data is safe (though I think it's more of a limitation than a feature).

Currently if you write an infinite loop in js, or write code in a way where it starts eating up memory we don't do anything to restart the js process, but it would be relatively easy to implement.

Re: RethinkDB: An open-source distributed database built with love over three years

#147

Post by StavrosK »

coffeemug wrote: post #146
StavrosK wrote: post #138

Earlier quoted context omitted.

By the way, how safe is the JS interpreter? Can you get into trouble by running untrusted code in map/reduce queries?

JS interpreter (V8 under the hood) runs in a process pool -- similar to a thread pool, but outside of the core rethinkdb process. The code running in the JS interpreter cannot corrupt memory or crash the rethinkdb process (if it crashes, rethinkdb will simply start another v8 process). You also can't write from js executed on the server, so the data is safe (though I think it's more of a limitation than a feature). C…

I see, thanks. I'm definitely interested in that, as I'm developing http://www.instahero.com and the current approach isn't very scalable, so I'm evaluating alternatives. RethinkDB looks like a good candidate so far.

Re: RethinkDB: An open-source distributed database built with love over three years

#148

Post by m0th87 »

coffeemug wrote: post #122
jdoliner wrote: post #116

Earlier quoted context omitted.

Piping all the data to the client would be extremely inefficient. Fortunately we don't do that. When a filter is written in the client language it gets compiled into a protocol buffer which is sent to the cluster. This gets compiled into a query which is sent to each of the relevant shards for the table. This query has the filter baked right into it. The shards then go through their local copy of the data and filter…

To add to jdoliner's answer, the reason why you can write table('foo').filter(lambda x: x['bar'] > 5).run() is because we do some language trickery on the client side to compile the query to an AST. In this case, we overload greater than operator, call the lambda function once on the client with a special object, and return an AST. This AST is then sent to the server and executed there. It's rather difficult to integ…

Does the function have to be referentially transparent?

Re: RethinkDB: An open-source distributed database built with love over three years

#149

Post by rgarcia »

m0th87 wrote: post #107
coffeemug wrote: post #103

Earlier quoted context omitted.

There is of course no fundamental reason why JSON-based db's has to be schemaless. This is one interesting direction that might be worth exploring.

I would love a system that is schema-less by design, but has guards that can be enforced at insert/update. That way, the underlying data structures don't have to be locked up from complex migrations (as needed w/ ALTER TABLE), but you still get type safety. A migration instead would simply involve a change in guards and an asynchronous update of existing entries. Plus you'd get all the wins of something resembling op…

At Clever we do this with mongo + mongoose. Mongoose is janky in places but it's great for type safety and migrations like you describe.
Post reply on HN