Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB 2.0 is now production ready

rethinkdb.com

Post reply on HN
Print view View original

151–156 of 156 posts

Re: RethinkDB 2.0 is now production ready

#151

Post by e12e »

danielmewes wrote: post #149
e12e wrote: post #134

Earlier quoted context omitted.

Rough numbers indeed - you forgot to define what a "server" is -- dedicated hw 16 core xeon with 4xssd in hw raid0 or a Digital Ocean vps with 512MB ram? ;-)

Daniel @ RethinkDB here. We'll release the details shortly. This was running on 12 core Xeon servers with 2 SSDs each in software RAID 0. There were also additional read queries running at the same time as the write queries, and the read throughput that coffeemug posted is the sustainable increase in reads/s that you get when adding an additional server to a cluster. Single-server performance is much higher due to mi…

10 gbps ethernet?

Re: RethinkDB 2.0 is now production ready

#152

Post by danielmewes »

lobster_johnson wrote: post #86
danielmewes wrote: post #78

Earlier quoted context omitted.

We did a couple of scalability improvements in 2.0, but didn't optimize groups and counts specifically. Would you mind writing me an email with your query or opening an issue at https://github.com/rethinkdb/rethinkdb/issues (unless you have already?)? I'd like to look into it to see how we can best improve this. We're planning to implemented a faster count algorithm that might help with this ( https://github.com/reth…

What I was doing is so trivial, you don't really need this information. This was my reference SQL query: select path, count(*) from posts group by path; (I don't have the exact Rethink query written down, but it was analogous to the SQL version.) You can demonstrate RethinkDB's performance issue with any largeish dataset by trying to group on a single field. The path column in this case has a cardinality of 94, and t…

As a second data point: I tried

    table.groupBy(function (x) { ... }).count()
where the function maps the documents into one out of 32 groups (so that's less than your 94, but shouldn't make a giant difference... I just had this database around). Did that on both 1 million and a 25 million document table, and memory usage looked fine and very stable. This was on RethinkDB 2.0, and I might retry that on 1.16 later to see if I can reproduce it there.

Do you remember if you had set an explicit cache size back when you were testing RethinkDB?

Re: RethinkDB 2.0 is now production ready

#153

Post by danielmewes »

e12e wrote: post #151
danielmewes wrote: post #149

Earlier quoted context omitted.

Daniel @ RethinkDB here. We'll release the details shortly. This was running on 12 core Xeon servers with 2 SSDs each in software RAID 0. There were also additional read queries running at the same time as the write queries, and the read throughput that coffeemug posted is the sustainable increase in reads/s that you get when adding an additional server to a cluster. Single-server performance is much higher due to mi…

10 gbps ethernet?

Two 1 GBit ethernet ports, one used for intra-cluster and one for client connections.

Re: RethinkDB 2.0 is now production ready

#154

Post by danielmewes »

cachvico wrote: post #120

Any thoughts about multi-doc transactions?

It's not currently on our road map.

Even though there are some well-researched algorithms for it, actually implementing transactions in a distributed system is pretty hard. It also comes at significant performance costs, which would interfere with our goal of easy and efficient scalability.

Re: RethinkDB 2.0 is now production ready

#155

Post by lobster_johnson »

danielmewes wrote: post #152
lobster_johnson wrote: post #86

Earlier quoted context omitted.

What I was doing is so trivial, you don't really need this information. This was my reference SQL query: select path, count(*) from posts group by path; (I don't have the exact Rethink query written down, but it was analogous to the SQL version.) You can demonstrate RethinkDB's performance issue with any largeish dataset by trying to group on a single field. The path column in this case has a cardinality of 94, and t…

As a second data point: I tried table.groupBy(function (x) { ... }).count() where the function maps the documents into one out of 32 groups (so that's less than your 94, but shouldn't make a giant difference... I just had this database around). Did that on both 1 million and a 25 million document table, and memory usage looked fine and very stable. This was on RethinkDB 2.0, and I might retry that on 1.16 later to se…

Cool. Well, the process eventually crashed if I used the defaults. I had to give it a 6GB cache (I think, maybe it was more) for it to return anything. The process would actually allocate that much, too, so it's clear that it was effectively loading everything into memory.

Re: RethinkDB 2.0 is now production ready

#156

Post by cachvico »

danielmewes wrote: post #154
cachvico wrote: post #120

Any thoughts about multi-doc transactions?

It's not currently on our road map. Even though there are some well-researched algorithms for it, actually implementing transactions in a distributed system is pretty hard. It also comes at significant performance costs, which would interfere with our goal of easy and efficient scalability.

Thank you for the comment. I was wondering if something along the lines of http://blog.labix.org/2012/08/22/multi-doc-transactions-for-... would be feasible.
Post reply on HN