Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB: An open-source distributed database built with love over three years

rethinkdb.com

Post reply on HN
Print view View original

221–230 of 247 posts

Re: RethinkDB: An open-source distributed database built with love over three years

#221

Post by lucian1900 »

coffeemug wrote: post #122
jdoliner wrote: post #116

Earlier quoted context omitted.

Piping all the data to the client would be extremely inefficient. Fortunately we don't do that. When a filter is written in the client language it gets compiled into a protocol buffer which is sent to the cluster. This gets compiled into a query which is sent to each of the relevant shards for the table. This query has the filter baked right into it. The shards then go through their local copy of the data and filter…

To add to jdoliner's answer, the reason why you can write table('foo').filter(lambda x: x['bar'] > 5).run() is because we do some language trickery on the client side to compile the query to an AST. In this case, we overload greater than operator, call the lambda function once on the client with a special object, and return an AST. This AST is then sent to the server and executed there. It's rather difficult to integ…

I find I prefer SQLAlchemy's Table.query.filter(Table.bar > 5) to a lambda that gets compiled to an AST in an odd way.

Re: RethinkDB: An open-source distributed database built with love over three years

#222

Post by ww520 »

Guillaume86 wrote: post #219
ww520 wrote: post #186

Earlier quoted context omitted.

Thanks for your and jdoliner's detail answers! Hope I didn't ask too many questions. :) I'll respond to both here. For 2 and 3, I think I didn't make it clear. Let me clarify. A common db problem with multiple clients is dealing with concurrent update on the same piece of data. E.g both client1 and client2 read D as D=15 at the same time. Client1 adds 1 to D as 16 and saves it. Then client2 adds 1 to D as 16 and save…

Your exemple of conditional update can be addressed using atomic update: r.table('tv_shows') .filter({ name: 'Star Trek TNG' }) .update({ episodes: r('episodes').add(1) }) .run() http://www.rethinkdb.com/docs/advanced-faq/#atomic

I think the atomicity model here works like a transaction on the whole document, where all the changes to the attributes of a document are updated all at once.

The scenario I described has to do with read-consistency, where the value read by a client should not be changed during the time of the read and the time of the update. The usual way of handling it was to take a write lock for the duration to prevent update from others but that degrades concurrency. The other way is to do optimistic lock (or conditional update) to allow the client to detect change during the time and retry with the new value.

Re: RethinkDB: An open-source distributed database built with love over three years

#223

Post by shrughes »

haberman wrote: post #220
erichocean wrote: post #140

Earlier quoted context omitted.

That sounds good, but with only 53 bits of integer precision in JSON (51 if you move the decimal point to account for cents), there's just not enough digits for finance these days. A similar problem exists if you use JSON numbers (aka doubles) for timestamps –- the numbers just aren't big enough to do it accurately.

JSON is not by definition limited to 53 bits of precision -- the standard itself does not specify range or precision. In practice, most implementations represent a JSON number with a double though.

Maybe one that wants to keep working through high inflation scenarios.

(But what financial software is running on a NoSQL database?)

Re: RethinkDB: An open-source distributed database built with love over three years

#224

Post by shrughes »

hcarvalhoalves wrote: post #210

It seems very interesting, and having to deal with ORMs daily makes me appreciate the clean API. I feel being based on JSON is a big con though. While it's popular, it was never meant to be a rich serialization format, just simple. How to implement more complex fields like dates, and query efficiently on RethinkDB?

A benefit to JSON is that people know immediately what it is. It certainly does have its deficiencies (binary data being one that outweighs dates in terms of immediate importance). Being limited to strict JSON is not a permanent decision (I'm saying this as a member of the RethinkDB engineering team), it's a conservative one in terms of API design, and in terms of limiting scope for the first release.

Re: RethinkDB: An open-source distributed database built with love over three years

#225

Post by coffeemug »

ww520 wrote: post #222
Guillaume86 wrote: post #219

Earlier quoted context omitted.

Your exemple of conditional update can be addressed using atomic update: r.table('tv_shows') .filter({ name: 'Star Trek TNG' }) .update({ episodes: r('episodes').add(1) }) .run() http://www.rethinkdb.com/docs/advanced-faq/#atomic

I think the atomicity model here works like a transaction on the whole document, where all the changes to the attributes of a document are updated all at once. The scenario I described has to do with read-consistency, where the value read by a client should not be changed during the time of the read and the time of the update. The usual way of handling it was to take a write lock for the duration to prevent update fr…

My point was that you don't have to do that with rethink because the entire query gets executed on the server. You don't have to take the value down to the client, make the change, and then send it back. The entire update gets evaluated on the server and the server handles atomicity in various ways (depending on the query).

Re: RethinkDB: An open-source distributed database built with love over three years

#226

Post by coffeemug »

lucian1900 wrote: post #221
coffeemug wrote: post #122

Earlier quoted context omitted.

To add to jdoliner's answer, the reason why you can write table('foo').filter(lambda x: x['bar'] > 5).run() is because we do some language trickery on the client side to compile the query to an AST. In this case, we overload greater than operator, call the lambda function once on the client with a special object, and return an AST. This AST is then sent to the server and executed there. It's rather difficult to integ…

I find I prefer SQLAlchemy's Table.query.filter(Table.bar > 5) to a lambda that gets compiled to an AST in an odd way.

You can do that too: r.table('foo').filter(r['bar'] > 5)

The use of r in filter is getting the attribute bar of the row.

Re: RethinkDB: An open-source distributed database built with love over three years

#228

Post by datasink »

mnutt wrote: post #204
adolfoabegg wrote: post #194

Earlier quoted context omitted.

Why would I switch to homebrew? I've been using Macports for 2 years now and I never had any issues with it. It works. Also migrating from Macports to homebrew (in case there's a good reason to do it) it would be a painful experience. I would have to start from scratch right?

Personally, I migrated to homebrew due to the ease of writing packages compared to macports. But I happen to enjoy ruby more than custom DSLs. It's probably a division similar to Chef/Puppet.

Having switched and then switched back, I will agree with you. If you're inclined to write your own package installers, Homebrew is worlds better than MacPorts. But otherwise, MacPorts has many more packages that-just-work.

Re: RethinkDB: An open-source distributed database built with love over three years

#229

Post by haberman »

shrughes wrote: post #223
haberman wrote: post #220

Earlier quoted context omitted.

JSON is not by definition limited to 53 bits of precision -- the standard itself does not specify range or precision. In practice, most implementations represent a JSON number with a double though.

Maybe one that wants to keep working through high inflation scenarios. (But what financial software is running on a NoSQL database?)

I think perhaps you replied to the wrong comment?

I didn't pose the question "what financial app needs >53 bits of precision?"

Re: RethinkDB: An open-source distributed database built with love over three years

#230

Post by getsat »

hyperbovine wrote: post #190
merlincorey wrote: post #166

Earlier quoted context omitted.

clang is better than gcc, IMO

clang either does not build or defectively builds certain things on OS X, for instance Ruby 1.9.3. I had to acquire vanilla GCC for this reason the other day, and was relieved to find it in HomeBrew.

Old-ass versions of Clang would build Ruby and PostgreSQL (client) binaries which would segfault upon execution. Try grabbing the latest XCode/command line tools package and you should be fine. I've been running Ruby 1.9.3 with Clang for quite a while.

brew install rbenv ruby-build; /* rc file shenanigans */; rbenv install 1.9.3-p327; rbenv global 1.9.3-p327; ruby --version

Post reply on HN