Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

rethinkdb.com

Post reply on HN
Print view View original

51–60 of 105 posts

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#51

Post by ddorian43 »

leif wrote: post #34
hendzen wrote: post #11

Earlier quoted context omitted.

Nope, operations on a single document are atomic, but not on multiple documents. MongoDB works the same way. AFAIK, the only NoSQL DB with support for multi-doc transactions is HyperDex [0]. [0] - http://hyperdex.org/

Our (Tokutek) build of MongoDB with Fractal Tree indexes has atomic multi-document updates: http://www.tokutek.com/2013/04/mongodb-transactions-yes/ Come talk to us if you want to give it a spin! Don't mean to steal the thunder from RethinkDB, they're doing great work too.

hyperdex says they have multi-server transaction while you still have single server (according to the comments on the blog post)

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#52

Post by mintplant »

dude_abides wrote: post #14
rdudekul wrote: post #4

I love RethinkDB API. It makes joins possible without going through mapreduce. Though I am using MongoDB currently, I plan on switching to RethinkDB as soon as it is more stable and provides better performance. Excited to see the team making progress on all fronts.

It makes joins possible without going through mapreduce. This is pure gold https://twitter.com/shit_hn_says material :)

...Compared to MongoDB, which is referred to in the very next sentence.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#53

Post by wpietri »

jdoliner wrote: post #13
hendzen wrote: post #5

Seems a bit unconventional to manually specify which index you'd like to use when writing queries. This would normally be something handled by a query optimizer. Of course, many DBMS provide some facility for specifying indices manually through query hinting, but for non-trivial queries, the optimizer often can do a better job than a programmer at picking the most efficient query plan.

Hi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on t…

Bless you.

The thing that makes me crazy about database engines is the "60% the time, it works every time" phenomenon, wherein things are magically fast until they are magically slow.

Then the discussion goes seamlessly from "don't worry your pretty head, just build your app" to "of course you have to really understand ." With no recognition that these are basically contradictory messages.

If you can really hide the complexity so it almost never matters, great. But here, where you couldn't, I think you made the right choice by exposing it.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#54

Post by mintplant »

dennis82 wrote: post #25

this is a good update and not to rain on their parade, but doesn't this seem market redundant and reinvent-the-wheel?

I'd rather have had them take their time to get it right than rush it out the door to be first-to-market.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#55

Post by LeafStorm »

> Early in the development of RethinkDB, we made the design decision to be extremely conservative about durability and safety of users's data.

I'm glad that the Rethink engineers are taking this approach to development instead of "let's win all the benchmarks!" Overall, it's apparent that they are really thinking through the consequences of their decisions and the tradeoffs in database systems, and that they want to build something that will last instead of something that will just become popular in the short term.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#56

Post by btilly »

sergei wrote: post #50
btilly wrote: post #44

Earlier quoted context omitted.

You are underestimating the necessary design tension. If you expose hints, then you've constrained how your optimizer works because it has to work with those hints. (Oracle, for example, has been fighting this battle for ages.) Once you've built an optimizer, people come to depend on it so it is critical that you get it right. They will need optimizer eventually, and they know it. But I'm glad that they are not imple…

I frequently see apps with thousands of distinct query signatures. Having a developer manually chose indexes, join ordering, aggregation method, (just to name a few) for every single query, and then select multiple plans because input parameters absolutely do result in scenario where plans can be 10,000x off in performance -- well, that's just untenable. Not trying to claim that CBOs are the panacea here, but let's b…

There is a logical fallacy in what you've said. It absolutely is true that there are times when input parameters can be a factor of 10k in performance. It is also absolutely true that applications can have thousands of distinct query signatures.

What DOES NOT follow is that most of those distinct query signatures are very important to the application. A few are. But most are not. However volunteering to add the wrong one can take the application down.

Having developers manually plan every single query is not the right choice.

I agree. However the many applications successfully built on MySQL demonstrate that for a lot of people, crappy but consistent planning is good enough. At least then people can know where the problems are, and fix them.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#57

Post by RyanZAG »

dude_abides wrote: post #14
rdudekul wrote: post #4

I love RethinkDB API. It makes joins possible without going through mapreduce. Though I am using MongoDB currently, I plan on switching to RethinkDB as soon as it is more stable and provides better performance. Excited to see the team making progress on all fronts.

It makes joins possible without going through mapreduce. This is pure gold https://twitter.com/shit_hn_says material :)

I'm not sure what you think is pure gold here - the statement is correct. When dealing with big data, joins are the first thing to go as you need to shard your data. Whether RethinkDB will really solve joins across sharded data is up in the air though - ultimately you still need to do filtering per shard and then reduce the result somewhere - and doing filtering across two attributes means you are going to be doing a lot of data transfer, but RethinkDB is about the only database I've seen really making progress in this area.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#58

Post by wmf »

RyanZAG wrote: post #57
dude_abides wrote: post #14

Earlier quoted context omitted.

It makes joins possible without going through mapreduce. This is pure gold https://twitter.com/shit_hn_says material :)

I'm not sure what you think is pure gold here - the statement is correct. When dealing with big data, joins are the first thing to go as you need to shard your data. Whether RethinkDB will really solve joins across sharded data is up in the air though - ultimately you still need to do filtering per shard and then reduce the result somewhere - and doing filtering across two attributes means you are going to be doing a…

Clustered databases are actually older than mapreduce, hence the hilarity.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#59

Post by btilly »

sixbrx wrote: post #48
btilly wrote: post #44

Earlier quoted context omitted.

You are underestimating the necessary design tension. If you expose hints, then you've constrained how your optimizer works because it has to work with those hints. (Oracle, for example, has been fighting this battle for ages.) Once you've built an optimizer, people come to depend on it so it is critical that you get it right. They will need optimizer eventually, and they know it. But I'm glad that they are not imple…

This accords with our experience with large Oracle databases as well. We've had a few panics caused by a DBA having updated table statistics when trying to optimize some query. This would occasionally cause sudden, massive changes in the way that other unrelated queries were performed, which queries would not finish and sometimes bring down the database. These experiences caused us to have to change our procedures an…

I wish that databases did "speculative" queries.

In the simplest form, you could say, "I have 2 reasonable plans, let's try A, and if it takes above time X, then start B in parallel and go with whatever finishes first."

You could ramp up the idea to handle changing query plans based on updated statistics by sending some fraction of queries to one plan, and some to another. Then keep stats on how that worked out for you.

Basically never simply flip the switch in production to try a new, unproven query.

Incidentally 12g advertises that they will actually collect statistics during query execution and based on those will validate the query plan. If that works, then this problem should get better. But of course that comes with overhead, and is likely to be a complex piece of code, so you tell me whether you trust them to get it right right away.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#60

Post by jdoliner »

wpietri wrote: post #53
jdoliner wrote: post #13

Earlier quoted context omitted.

Hi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on t…

Bless you. The thing that makes me crazy about database engines is the "60% the time, it works every time" phenomenon, wherein things are magically fast until they are magically slow. Then the discussion goes seamlessly from "don't worry your pretty head, just build your app" to " of course you have to really understand ." With no recognition that these are basically contradictory messages. If you can really hide the…

Thanks, we've gotten a lot of mileage out of the following ethos: Let users ignore danger, but never let users ignore the fact that they're ignoring danger. Always give users a way to learn exactly what the dangers are.

In the case of the query optimizer we haven't gotten to giving people a way to ignore these dangers yet but we will eventually.

Post reply on HN