Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

rethinkdb.com

Post reply on HN
Print view View original

31–40 of 105 posts

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#31

Post by coffeemug »

hendzen wrote: post #22
jdoliner wrote: post #13

Earlier quoted context omitted.

Hi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on t…

Thanks for the reply. Do you have any resources you could share regarding the pitfalls of query optimizers? My experience with them comes primarily from studying the System R optimizer where the literature presented query optimization as a boon to performance without mentioning such drawbacks. Congratulations on the release by the way.

The primary problem is that picking indexes and execution algorithms essentially involves traversing an exponential space (and taking guesses about costs of things). Modern DB optimizers have hundreds (or thousands) of heuristics to do this well, but every once in a while they pick the wrong path and run a suboptimal query. This might be ok for offline analytics, but can be disastrous for production OLTP environments because a small change in statistical information can trip up such an edge case and the live system will crawl to a halt. For example, Postgres optimizer has been around for a long time and is very mature, but they still fix these bugs quite often AFAIK.

In many production environments admins end up having to hack the queries to "trick" the optimizer into doing what they want, which of course defeats the whole purpose. So, for real-time systems, being able to specify indexes manually is actually a productivity boost, because you often know exactly how you want the query executed.

I'll try to dig up some info on this, I don't have any links off the top of my head.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#32

Post by sergei »

jdoliner wrote: post #13
hendzen wrote: post #5

Seems a bit unconventional to manually specify which index you'd like to use when writing queries. This would normally be something handled by a query optimizer. Of course, many DBMS provide some facility for specifying indices manually through query hinting, but for non-trivial queries, the optimizer often can do a better job than a programmer at picking the most efficient query plan.

Hi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on t…

A decent cost based optimizer will be right most of the time, and you have hints for the cases where it falters.

Your approach is simply hiding the query optimizer choices inside of the app, making it more fragile.

Let's say I have two fields: a and b, both of which have an index. And then I have a query with predicates across both a and b. For example, a = 10 and b = 20.

One of the core facilities within an optimizer is selectivity estimation. By looking at the statistics, the optimizer will see that a = 10 might look at 10,000 rows while b = 20 might only look at 10. So the optimal and desired choice would be to use index over b.

However, the exact same query construct with different parameters (e.g. a = 50 and b = 3) might flip the index selection.

Now let's imagine I had to implement this inside of my app. Every time I have such a constraint, the app has to become aware of selectivity to know which indexes to use based on input parameters.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#33

Post by tomjen3 »

prodigal_erik wrote: post #16
hendzen wrote: post #11

Earlier quoted context omitted.

Nope, operations on a single document are atomic, but not on multiple documents. MongoDB works the same way. AFAIK, the only NoSQL DB with support for multi-doc transactions is HyperDex [0]. [0] - http://hyperdex.org/

Also http://www.rethinkdb.com/docs/advanced-faq/ . RethinkDB started out as a MySQL driver that was lock-free and did schema updates non-stupidly, but it sounds like they've abandoned ACID and schema enforcement entirely like almost all NoSQL systems (as if what I wanted is faster wrong answers).

I see this tude' among DB guys quite often -- unless the database guarantees unconditionally that it will always give the right answer and that no data is ever lost then it is worse than useless and most be nuked from orbit.

Quite frankly that is pretty arrogant. ACID may be required for a banks transactions, but in far the majority of cases it isn't required (yes, even in your business) -- nothing of any consequence is going to happen if your A/B testing DB lose a couple houndred entries because those weren't flushed to the cache.

As for non-schema enforcement, well you have too add a few if statements to your database model class -- and that is it.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#34

Post by leif »

hendzen wrote: post #11
dscrd wrote: post #9

Does RethinkDB do multiple atomic updates?

Nope, operations on a single document are atomic, but not on multiple documents. MongoDB works the same way. AFAIK, the only NoSQL DB with support for multi-doc transactions is HyperDex [0]. [0] - http://hyperdex.org/

Our (Tokutek) build of MongoDB with Fractal Tree indexes has atomic multi-document updates: http://www.tokutek.com/2013/04/mongodb-transactions-yes/ Come talk to us if you want to give it a spin!

Don't mean to steal the thunder from RethinkDB, they're doing great work too.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#35

Post by josegonzalez »

sergei wrote: post #32
jdoliner wrote: post #13

Earlier quoted context omitted.

Hi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on t…

A decent cost based optimizer will be right most of the time, and you have hints for the cases where it falters. Your approach is simply hiding the query optimizer choices inside of the app, making it more fragile. Let's say I have two fields: a and b, both of which have an index. And then I have a query with predicates across both a and b. For example, a = 10 and b = 20. One of the core facilities within an optimize…

Which is good, because now your developers don't just add queries to the application without thinking about performance.

You don't have to be a dba to understand how using one index vs another will affect the performance - and thus conversion rate - of your application.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#36

Post by jdoliner »

coffeemug wrote: post #21
jdoliner wrote: post #17

Earlier quoted context omitted.

That's something we'll make sure to keep around indefintely. I always hate things that are too magically, which optimizers frequently seem to me, but I also get why they're useful to people. The best compromise I think is to give people the magic and a way to opt out of the magic.

The best compromise I think is to give people the magic and a way to opt out of the magic. This is actually a surprisingly succinct way to describe the philosophy behind most RethinkDB features. This approached worked really well so far.

Get thee to a tweetery.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#37

Post by mfenniak »

jdoliner wrote: post #13
hendzen wrote: post #5

Seems a bit unconventional to manually specify which index you'd like to use when writing queries. This would normally be something handled by a query optimizer. Of course, many DBMS provide some facility for specifying indices manually through query hinting, but for non-trivial queries, the optimizer often can do a better job than a programmer at picking the most efficient query plan.

Hi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on t…

Hi Joe,

I just wanted to pipe in and say that's excellent reasoning for this design, especially since the ability to create indexes on arbitrary ReQL expressions would make an intelligent query planner into a lot of work. I'm looking forward to giving these a try.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#38

Post by jdoliner »

apendleton wrote: post #30
papsosouid wrote: post #27

Wow, I thought this was a joke press release making fun of "web scale" fad craziness. They seriously gave a 1.0 release number to a "database" that you couldn't search?

No, you've always been able to execute queries. They just weren't indexed before.

Data has always been indexed it just was only indexed by a single primary key. Before this release the indexing scheme wasn't more sophisticated than a key value store now it is. I feel like there are several other products on the market that just do key value indexing.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#39

Post by papsosouid »

apendleton wrote: post #30
papsosouid wrote: post #27

Wow, I thought this was a joke press release making fun of "web scale" fad craziness. They seriously gave a 1.0 release number to a "database" that you couldn't search?

No, you've always been able to execute queries. They just weren't indexed before.

I didn't say "execute queries", I said search. Unless you have a tiny amount of data, no indexes means searching is not feasible.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#40

Post by dennis82 »

josegonzalez wrote: post #35
sergei wrote: post #32

Earlier quoted context omitted.

A decent cost based optimizer will be right most of the time, and you have hints for the cases where it falters. Your approach is simply hiding the query optimizer choices inside of the app, making it more fragile. Let's say I have two fields: a and b, both of which have an index. And then I have a query with predicates across both a and b. For example, a = 10 and b = 20. One of the core facilities within an optimize…

Which is good, because now your developers don't just add queries to the application without thinking about performance. You don't have to be a dba to understand how using one index vs another will affect the performance - and thus conversion rate - of your application.

no, it's categorically not good to make an application more fragile. A weakness restated is not a strength. Every database should have an optimizer, period.
Post reply on HN