Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB: An open-source distributed database built with love over three years

rethinkdb.com

Post reply on HN
Print view View original

211–220 of 247 posts

Re: RethinkDB: An open-source distributed database built with love over three years

#211

Post by erichocean »

gruseom wrote: post #205
erichocean wrote: post #172

Earlier quoted context omitted.

Sure, and I'm not trying to imply the RethinkDB guys are writing shoddy code or anything. For all I know the thing is bug-free with fantastic performance, perfect linear scaling with both number of cores and number of nodes in the cluster, and really does let you run your analytic workload on the same cluster you're taking transactions on (though I really doubt this last one – running analytics on your transactional…

You say "yeah, that's straightforward to do" and also you "really doubt" that their claims are true? Reminds me of Freud's story about the peasant who says to another, "Hey, you broke that kettle I lent you", and the other says, "It was fine when I gave it back to you, it was already broken when you lent it to me, and I never borrowed it."

Running analytics on a database is both "really straightforward to do" and at the same time, I "really doubt" that anyone would actually do both analytics and transactions on the same database instance in production.

Why? Analytics are CPU hogs, tend to access tons of data in random fashion (blowing caches and hogging the SSD drive), and given that RethinkDB has no secondary indexing, are likely to be especially slow.

That's why people have separate machines dedicated to analytics. What I think a team would actually do with RethinkDB is the same thing people do with Cassandra: include a separate cluster (in the same or a remote datacenter) and replicate data to it from the transactional cluster(s). They would then run analytics on the analytics cluster.

This approach won't impact transactional latency, and also allows you to have different hardware altogether for running analytics (e.g. tons of cores and RAM that might go wasted on the transactional DB machines).

This is all Big Data 101; it's not controversial.

Re: RethinkDB: An open-source distributed database built with love over three years

#212

Post by erichocean »

Nitramp wrote: post #193
coffeemug wrote: post #74

Earlier quoted context omitted.

I mostly agree with what you're saying, but I also think there's enormous value in making easy things be really easy. Even with today's state of the art adding a shard, dealing with consistency issues, adding replicas, etc. is relatively hard. Perhaps not in a computer-sciency sense (all the problems are fairly well understood), but in an operational sense. Lots and lots of work needs to be done even with systems lik…

there are tougher problems like high-performance cross-document distributed ACID, but I think the industry as a whole is relatively far away from that right now Megastore and Spanner solve that problem, with varying tradeoffs: http://research.google.com/pubs/pub36971.html http://research.google.com/archive/spanner.html

Our internal database does too (with a different design than Spanner, but stuff still comes "online" atomically for everyone across the globe at the same time, with similar latency). Unlike FoundationDB, and like Spanner, we're doing it with complex object graphs, not just key-values, and we also do it with consistent secondary indexing (I'm not sure if Spanner supports this or not).

This isn't "the future", this is now. People are doing it, and have been for awhile. If you're going to "rethink the database", distributed global consistency should be at the top of your list today. RethinkDB seems like its merely "rethinking" Mongo.

The main benefit of global consistency, of course, is ease of use. Global consistency is so much easier to reason about and write code for!

Re: RethinkDB: An open-source distributed database built with love over three years

#213

Post by johncoltrane »

i386 wrote: post #159
biturd wrote: post #153

Earlier quoted context omitted.

Thanks for this work, it looks really nice. I was looking at the github comments about a home brew recipe in which it was stated that aside from a recipe creating a VM, the Mac OS X port would take a bit longer. Is that a full port from one language to another? Or just an issue of the different flavors of *nix that need dealing with and probably some of the dependency tree issues that come with it? I'm curious what n…

Seems to me that most of us who have used MacPorts have moved to Homebrew or that could just be the bubble I'm living in. Is there anyone still who still uses MacPorts who could chime in and say why they never made the switch?

I never did the switch because I never had to complain about MacPorts. Last time I looked at Homebrew it had a very small number of receipes compared to MacPorts' ports.

I ran a fairly convincing "Linux-like" alternative desktop using Awesome in XQuartz for about a year before I switched to Linux full time. That was before Homebrew but I'm quite certain that it would have been impossible with it.

Re: RethinkDB: An open-source distributed database built with love over three years

#214

Post by jdoliner »

coffeemug wrote: post #168
m0th87 wrote: post #148

Earlier quoted context omitted.

Does the function have to be referentially transparent?

No. E.g., you could write r.table('foo').update(lambda row: row.merge({'bar': row['bar'] + 1 })). A shortcut for this is r.table('foo').update({'bar': r['bar'] + 1 }). Neither is referentially transparent, and both work.

I believe both of those functions are referential transparent because they're pure functions. An example of a non referential transparent function would be:

r.table('foo').update(lambda row: {'bar' : r.table('bar').get(row["bar_id"]))

This still works but gets evaluated in a different way to make sure every secondary winds up with the same value.

Re: RethinkDB: An open-source distributed database built with love over three years

#215

Post by i386 »

hyperbovine wrote: post #190
merlincorey wrote: post #166

Earlier quoted context omitted.

clang is better than gcc, IMO

clang either does not build or defectively builds certain things on OS X, for instance Ruby 1.9.3. I had to acquire vanilla GCC for this reason the other day, and was relieved to find it in HomeBrew.

Ruby 1.9.3 working fine here and built with clang. What was defective?

Re: RethinkDB: An open-source distributed database built with love over three years

#216

Post by gruseom »

erichocean wrote: post #211
gruseom wrote: post #205

Earlier quoted context omitted.

You say "yeah, that's straightforward to do" and also you "really doubt" that their claims are true? Reminds me of Freud's story about the peasant who says to another, "Hey, you broke that kettle I lent you", and the other says, "It was fine when I gave it back to you, it was already broken when you lent it to me, and I never borrowed it."

Running analytics on a database is both "really straightforward to do" and at the same time, I "really doubt" that anyone would actually do both analytics and transactions on the same database instance in production. Why? Analytics are CPU hogs, tend to access tons of data in random fashion (blowing caches and hogging the SSD drive), and given that RethinkDB has no secondary indexing, are likely to be especially slow…

The RethinkDB guys made it clear in this thread that although they don't have secondary indexes in this release, they will definitely be adding them. They also explained why.

Re: RethinkDB: An open-source distributed database built with love over three years

#217

Post by coffeemug »

StavrosK wrote: post #208
coffeemug wrote: post #199

Earlier quoted context omitted.

Oh yes!

Wanna get a game going sometime? I play DOTA 2, by the way, hopefully you do too.

We have at least three dota players in the office. We should ge this going!

Re: RethinkDB: An open-source distributed database built with love over three years

#218

Post by coffeemug »

willytobler wrote: post #206

This looks like very interesting. And a nice interface to deal with. But ... I do not find anything about authentication. Whoever wants can fiddle with this 8080 Port. Did you rethinkAuth or am I just too stupid to RTFM?

No authentication support yet unfortunately - you'd have to do an ssh tunnel for the web admin. This is one of the things on the todo list.

Re: RethinkDB: An open-source distributed database built with love over three years

#219

Post by Guillaume86 »

ww520 wrote: post #186
coffeemug wrote: post #182

Earlier quoted context omitted.

1. Yes. It's a matter of doing this right, which will take some time. 2. Yes. There is no special command, you just combine update and branch ( http://www.rethinkdb.com/api/#py:control_structures-branch ) Here's an example in Python: r.table('foo').get(5).update({ 'bar': r.branch(r['baz'] == 0, 1, 2)}) This will set attribute bar to 1 if baz is 0, or to two 2 otherwise. Everything is atomic on that document. 3. Curre…

Thanks for your and jdoliner's detail answers! Hope I didn't ask too many questions. :) I'll respond to both here. For 2 and 3, I think I didn't make it clear. Let me clarify. A common db problem with multiple clients is dealing with concurrent update on the same piece of data. E.g both client1 and client2 read D as D=15 at the same time. Client1 adds 1 to D as 16 and saves it. Then client2 adds 1 to D as 16 and save…

Your exemple of conditional update can be addressed using atomic update:

r.table('tv_shows') .filter({ name: 'Star Trek TNG' }) .update({ episodes: r('episodes').add(1) }) .run()

http://www.rethinkdb.com/docs/advanced-faq/#atomic

Re: RethinkDB: An open-source distributed database built with love over three years

#220

Post by haberman »

erichocean wrote: post #140
cuu508 wrote: post #137

Earlier quoted context omitted.

Or you could use fixed point math: 600 in database means $6.00 Then aggregates and comparison operators would work, but you would have to decide upfront how much precision you might ever need

That sounds good, but with only 53 bits of integer precision in JSON (51 if you move the decimal point to account for cents), there's just not enough digits for finance these days. A similar problem exists if you use JSON numbers (aka doubles) for timestamps –- the numbers just aren't big enough to do it accurately.

JSON is not by definition limited to 53 bits of precision -- the standard itself does not specify range or precision. In practice, most implementations represent a JSON number with a double though.
Post reply on HN