Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

rethinkdb.com

Post reply on HN
Print view View original

61–70 of 105 posts

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#61

Post by jdoliner »

LeafStorm wrote: post #55

> Early in the development of RethinkDB, we made the design decision to be extremely conservative about durability and safety of users's data. I'm glad that the Rethink engineers are taking this approach to development instead of "let's win all the benchmarks!" Overall, it's apparent that they are really thinking through the consequences of their decisions and the tradeoffs in database systems, and that they want to…

Thanks. If we can succeed in being Mongo without the gotchas we'll be very happy.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#62

Post by leif »

ddorian43 wrote: post #51
leif wrote: post #34

Earlier quoted context omitted.

Our (Tokutek) build of MongoDB with Fractal Tree indexes has atomic multi-document updates: http://www.tokutek.com/2013/04/mongodb-transactions-yes/ Come talk to us if you want to give it a spin! Don't mean to steal the thunder from RethinkDB, they're doing great work too.

hyperdex says they have multi-server transaction while you still have single server (according to the comments on the blog post)

Yep, we aren't doing sharded transactions yet. Still trying to nail down the basics.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#63

Post by hendzen »

btilly wrote: post #59
sixbrx wrote: post #48

Earlier quoted context omitted.

This accords with our experience with large Oracle databases as well. We've had a few panics caused by a DBA having updated table statistics when trying to optimize some query. This would occasionally cause sudden, massive changes in the way that other unrelated queries were performed, which queries would not finish and sometimes bring down the database. These experiences caused us to have to change our procedures an…

I wish that databases did "speculative" queries. In the simplest form, you could say, "I have 2 reasonable plans, let's try A, and if it takes above time X, then start B in parallel and go with whatever finishes first." You could ramp up the idea to handle changing query plans based on updated statistics by sending some fraction of queries to one plan, and some to another. Then keep stats on how that worked out for y…

I'm sure you'll appreciate the irony of this, but what you described is pretty much the exact mechanism of the MongoDB query optimizer [0].

[0] - http://docs.mongodb.org/manual/core/read-operations/#query-o...

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#64

Post by Ecio78 »

Congrats for your release, RethinkDB seems very interesting and I really appreciate how coffeemug and all the other team members genuinely and clearly reply to questions here.

Is there already a list of sites/services that use RethinkDB in production?

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#65

Post by AlisdairO »

btilly wrote: post #59
sixbrx wrote: post #48

Earlier quoted context omitted.

This accords with our experience with large Oracle databases as well. We've had a few panics caused by a DBA having updated table statistics when trying to optimize some query. This would occasionally cause sudden, massive changes in the way that other unrelated queries were performed, which queries would not finish and sometimes bring down the database. These experiences caused us to have to change our procedures an…

I wish that databases did "speculative" queries. In the simplest form, you could say, "I have 2 reasonable plans, let's try A, and if it takes above time X, then start B in parallel and go with whatever finishes first." You could ramp up the idea to handle changing query plans based on updated statistics by sending some fraction of queries to one plan, and some to another. Then keep stats on how that worked out for y…

You can accomplish something pretty similar to this with oracle plan management: http://docs.oracle.com/cd/B28359_01/server.111/b28274/optpla... . You can set this up to pin plans, but let you know when it thinks it has a better plan to try out.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#66

Post by coffeemug »

Ecio78 wrote: post #64

Congrats for your release, RethinkDB seems very interesting and I really appreciate how coffeemug and all the other team members genuinely and clearly reply to questions here. Is there already a list of sites/services that use RethinkDB in production?

Hi :)

There isn't a list yet we can share, but we're starting to work very closely with a small group of early customers -- shoot me an email to slava@rethinkdb.com if you'd like to see if we can work together!

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#67

Post by JulianMorrison »

jdoliner wrote: post #13
hendzen wrote: post #5

Seems a bit unconventional to manually specify which index you'd like to use when writing queries. This would normally be something handled by a query optimizer. Of course, many DBMS provide some facility for specifying indices manually through query hinting, but for non-trivial queries, the optimizer often can do a better job than a programmer at picking the most efficient query plan.

Hi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on t…

Deterministic semantics are something I approve of. I'd rather have indexes under my control than the DB's.

Is it one index per chained operation (filter with index A then join with index B, for example)? Or can you only use one index per complete query?

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#68

Post by jdoliner »

JulianMorrison wrote: post #67
jdoliner wrote: post #13

Earlier quoted context omitted.

Hi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on t…

Deterministic semantics are something I approve of. I'd rather have indexes under my control than the DB's. Is it one index per chained operation (filter with index A then join with index B, for example)? Or can you only use one index per complete query?

You can use multiple indexes in some queries. The case you described will work. However you can't use multiple indexes from the same table. For example:

  table.between(...).between(...)
won't work. You'd have to use a filter (and thus a linear scan to do the second between. This seems like it would be a good thing for us to add to the FAQ because the rules on it can be a bit complicated. However the best way to find out might actually just be to try. Because we're not using an optimizer in the background you have to explicitly say which indexes you're using and thus they system will just refuse to do queries if it can't use indexes the way you're asking it to.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#69

Post by coffeemug »

JulianMorrison wrote: post #67
jdoliner wrote: post #13

Earlier quoted context omitted.

Hi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on t…

Deterministic semantics are something I approve of. I'd rather have indexes under my control than the DB's. Is it one index per chained operation (filter with index A then join with index B, for example)? Or can you only use one index per complete query?

You can't currently do index intersection (so you can't chain `between` commands), but you can absolutely use multiple indexes with joins. E.g. `table.between(...).eq_join(...).eq_join(...)` will use three indexes in this example.

EDIT: err, Joe beat me to it.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#70

Post by [deleted] »

btilly wrote: post #59
sixbrx wrote: post #48

Earlier quoted context omitted.

This accords with our experience with large Oracle databases as well. We've had a few panics caused by a DBA having updated table statistics when trying to optimize some query. This would occasionally cause sudden, massive changes in the way that other unrelated queries were performed, which queries would not finish and sometimes bring down the database. These experiences caused us to have to change our procedures an…

I wish that databases did "speculative" queries. In the simplest form, you could say, "I have 2 reasonable plans, let's try A, and if it takes above time X, then start B in parallel and go with whatever finishes first." You could ramp up the idea to handle changing query plans based on updated statistics by sending some fraction of queries to one plan, and some to another. Then keep stats on how that worked out for y…

[deleted]
Post reply on HN