Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB 2.0 is now production ready

rethinkdb.com

Post reply on HN
Print view View original

51–60 of 156 posts

Re: RethinkDB 2.0 is now production ready

#51

Post by lobster_johnson »

coffeemug wrote: post #29
mping wrote: post #25

Anyone has some numbers on performance? I tried RethinkDB 1.x and the performance wasn't quite there yet, specially bulk import and aggregations.

We'll be publishing a performance report soon (we didn't manage to get it out today). Rough numbers you can expect for 1KB size documents, 25M document database: 40K reads/sec/server, 5K writes/sec/server, roughly linear scalability across nodes. We should be able to get the report out in a couple of days.

Any work done in 2.0 for improving aggregation performance?

The last time I tried with 1.16, I gave up my testing when even the simplest aggregation query (count + group by with what should be a sequential, streaming scan) took literally minutes with RethinkDB, compared to <1s with PostgreSQL. Rethink coredumped before I gave it enough RAM, after which it blew up to around 7GB, whereas Postgres uses virtually no RAM, mostly OS buffers.

Re: RethinkDB 2.0 is now production ready

#52

Post by dorfsmay »

coffeemug wrote: post #28
dorfsmay wrote: post #21

Does RethinDB has a concept of transaction? My question is actually about restoring a lost node... If a node is rebooted, will all the data for its shards going to be sent again? Or just the delta? Similarly if I have to rebuild a node from scratch, is there a way to prime it so that a massive copy of all the data in the cluster gets copied to it from the other nodes?

> If a node is rebooted, will all the data for its shards going to be sent again? Or just the delta? Just the delta. We built an efficient, distributed BTree diff algorithm. When a node goes offline and comes back up, the cluster only sends a diff that the node missed. > Similarly if I have to rebuild a node from scratch, is there a way to prime it so that a massive copy of all the data in the cluster gets copied to…

> You don't have to do that, it happens automatically

Well, in a past life, I used another store that did that automatically, the issue with that is that EITHER it kills the cluster because of read-congestion as it re-builds the "new" node, OR, if you limit the bandwidth for node-building, it takes for ever and a half to rebuild a node which means that you are exposed with one less shard of what was on that node.

What are the chances of a filesystem snapshot to be consistent enough to be used to prime a crashed node? What about restoring backup files from other nodes?

Re: RethinkDB 2.0 is now production ready

#53

Post by mmrmissing »

jwr wrote: post #49
coffeemug wrote: post #24

Earlier quoted context omitted.

Slava @ Rethink here. We're planning to take the most well-supported community drivers under the RethinkDB umbrella (assuming the authors agree, of course). It will almost certainly be a collaboration with the community, but we'll be contributing much more to the community drivers, supporting the authors, and offering commercial support for these drivers to our customers.

Very glad to hear that. I tried using RethinkDB with Clojure recently, but there are two drivers. Both are mentioned on your pages. Figuring out which driver I should use isn't a great start — so even if you don't do a lot of development, just pointing to the drivers you consider "canonical" would help.

I had to create a small project for programming class. I settled on clojure and revise(bitemyapp's driver). Avoid this one. I hit on driver bug today, but didn't have the time to fix/report it. Just switched to the other one...

Re: RethinkDB 2.0 is now production ready

#54

Post by dkhenry »

Awesome news. I have used Rethink for a few internal projects and while I don't think it has that one "killer feature" that other DB's don't it is such a painless experience in development and deployment that makes just worlds better then trying to set up and scale some of the other solutions.

BZ rethinkdb team.

Re: RethinkDB 2.0 is now production ready

#55

Post by Shamanmuni »

vezzy-fnord wrote: post #10
danielmewes wrote: post #6

Earlier quoted context omitted.

Daniel @ RethinkDB here. As you mention, RethinkDB is fully open source so RethinkDB is always going to remain freely available.

This does not discount the possibility of a license change after a hypothetical acquisition, however. Though in that case you'll likely get a community fork branching off the upstream proprietary base.

No, the AGPL uses strong copyleft, so any future derivative work must be released under the same terms (and the same license or later versions if I'm not mistaken). The only possibility is to start a closed source clone that doesn't use any of the original code from zero.

The cases in which the community forks a project licensed with a copyleft license (like LibreOffice) has to do with insatisfaction with the direction in which the company that owns the original trademarks is leading said project. There's no risk of closing the source code.

Re: RethinkDB 2.0 is now production ready

#56

Post by coffeemug »

dorfsmay wrote: post #52
coffeemug wrote: post #28

Earlier quoted context omitted.

> If a node is rebooted, will all the data for its shards going to be sent again? Or just the delta? Just the delta. We built an efficient, distributed BTree diff algorithm. When a node goes offline and comes back up, the cluster only sends a diff that the node missed. > Similarly if I have to rebuild a node from scratch, is there a way to prime it so that a massive copy of all the data in the cluster gets copied to…

> You don't have to do that, it happens automatically Well, in a past life, I used another store that did that automatically, the issue with that is that EITHER it kills the cluster because of read-congestion as it re-builds the "new" node, OR, if you limit the bandwidth for node-building, it takes for ever and a half to rebuild a node which means that you are exposed with one less shard of what was on that node. Wha…

Congestion vs. time is definitely a hard problem. We've done an enormous amount of tuning to make this work, and the upcoming Raft release does even more. This part has been quite solid for a while, so I think you might have a better experience with RethinkDB than what you're used to.

There is currently no other way to prime the node -- I hope we don't have to add it. This sort of functionality should work out of the box.

Re: RethinkDB 2.0 is now production ready

#57

Post by ScottBurson »

coffeemug wrote: post #26
dorfsmay wrote: post #20

Why would I use RethinkDB instead of OrientDB?

Check out http://rethinkdb.com/faq/ for details on when RethinkDB is a great choice. The short version is that if you're building realtime apps, RethinkDB is an awesome choice because it pushes data to the application (which makes building and scaling realtime apps dramatically easier).

Hi Slava, the FAQ has a typo in the second sentence: "architecutre".

Re: RethinkDB 2.0 is now production ready

#58

Post by sixdimensional »

coffeemug wrote: post #46
notdonspaulding wrote: post #42

Cool! I've started to look into RethinkDB in the past, and I'm very interested in the features it claims. However, I only have so much time to investigate new primary storage solutions, and our team has been burned in the past by jumping too quickly on a DB's bandwagon when the reliability, performance, or tooling just wasn't there. As of late, we've come to rely on Aphyr's wonderful Call Me Maybe series[0] as a guid…

Slava @ Rethink here. This is a great point, and we're on it! We have a Raft implementation that unfortunately didn't make it into 2.0 (these things require an enormous amount of patient testing). The implementation is designed explicitly to support robust automatic failover, no interruptions during resharding, and all the edge cases exposed in the Jepsen tests (and many issues that aren't). This should be out in a f…

This conservative/consistent/responsible approach is one of the reasons I have faith in RethinkDB. You always seem to be taking the time to build it right and that is priceless.

Re: RethinkDB 2.0 is now production ready

#59

Post by coffeemug »

ScottBurson wrote: post #57
coffeemug wrote: post #26

Earlier quoted context omitted.

Check out http://rethinkdb.com/faq/ for details on when RethinkDB is a great choice. The short version is that if you're building realtime apps, RethinkDB is an awesome choice because it pushes data to the application (which makes building and scaling realtime apps dramatically easier).

Hi Slava, the FAQ has a typo in the second sentence: "architecutre".

Thanks -- fixed. Will take a little bit to push the site update live.

Re: RethinkDB 2.0 is now production ready

#60

Post by xtrumanx »

Lots of congratulating on this thread and a hell of a lot of points for a software release. I've been on HN consistently for a long while and I didn't realize there was so much love and hype for RethinkDB here.

Have I missed something?

Post reply on HN