Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

rethinkdb.com

Post reply on HN
Print view View original

91–100 of 105 posts

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#91

Post by hoodoof »

coffeemug wrote: post #90
hoodoof wrote: post #89

Earlier quoted context omitted.

The basic stuff needed to let a beginner developer write a web app should be super super simple. REALLY simple. No map no reduce nothing hard. Here are some of the basic examples of typical things people might need to do when writing a web app: How do I select records based on a value in an array? How would I find all records whose name begins with "star" i.e. 'Star Trek' 'Star Base' 'Star Wars' 'Star Anise' I have r…

Thank you, this is phenomenal feedback!! Most of these are already very simple, and the ones that aren't will become very simple soon. Michel (@neumino on github) is working on improving the docs right now, and I will pass this list on to him. We'll get this documented ASAP. Thank you again, this is incredibly helpful!

I look forward to returning to RethinkDB. Great product and the developers are so friendly and helpful. Good attitude.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#93

Post by Game_Ender »

craigyk wrote: post #92
agildehaus wrote: post #8

Build fails on a "brew upgrade" for me. http://pastebin.com/3GWetUTE

It's not always 100%, but who else think homebrew is awesome? I miss it when I go to my linux machines.

What do you miss about it?

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#94

Post by nickpresta »

craigyk wrote: post #92
agildehaus wrote: post #8

Build fails on a "brew upgrade" for me. http://pastebin.com/3GWetUTE

It's not always 100%, but who else think homebrew is awesome? I miss it when I go to my linux machines.

I like Homebrew as well and I don't want to detract from how awesome it is, but dpkg + aptitude is much better in my opinion and I find I actually miss aptitude on OSX.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#95

Post by anatari »

So is this production ready? Excited about this, but I don't have time at the moment to dabble on things I can't use and the last comment regarding this is a post on Quora from the founder saying that it is not production ready yet. 1.0+ version numbers makes you think this isn't beta software anymore, but that doesn't seem to be the case.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#96

Post by jeffdavis »

hendzen wrote: post #63
btilly wrote: post #59

Earlier quoted context omitted.

I wish that databases did "speculative" queries. In the simplest form, you could say, "I have 2 reasonable plans, let's try A, and if it takes above time X, then start B in parallel and go with whatever finishes first." You could ramp up the idea to handle changing query plans based on updated statistics by sending some fraction of queries to one plan, and some to another. Then keep stats on how that worked out for y…

I'm sure you'll appreciate the irony of this, but what you described is pretty much the exact mechanism of the MongoDB query optimizer [0]. [0] - http://docs.mongodb.org/manual/core/read-operations/#query-o...

A cost-based optimizer also needs a way to handle the combinatorial explosion of possible plans. For MongoDB, maybe they can be exhaustive or use a simple strategy. But for a SQL DBMS, the search space is way too huge, and you need a way to navigate it intelligently.

This "try the plan out" idea has come up many times, and it's a reasonable idea, but it's nothing new and not a complete solution. You need a way to evaluate whether a plan has a reasonable chance of being the best one before the whole plan is even constructed, much less executed.

Combinatorial explosion is also one of the reasons that humans generally aren't great at coming up with non-trivial plans, either.

A good optimizer is a combination of many good ideas; so I don't mean to be discouraging, just that one idea is not the full story.

If you are worried about the risk of a changing plan, you are between a rock and a hard place. If you lock the plan down (or hard-code it in the application), and the data characteristics change, that could be disastrous. If you don't lock it down, that's a leap of faith that the optimizer will adapt gracefully.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#97

Post by hendzen »

jeffdavis wrote: post #96
hendzen wrote: post #63

Earlier quoted context omitted.

I'm sure you'll appreciate the irony of this, but what you described is pretty much the exact mechanism of the MongoDB query optimizer [0]. [0] - http://docs.mongodb.org/manual/core/read-operations/#query-o...

A cost-based optimizer also needs a way to handle the combinatorial explosion of possible plans. For MongoDB, maybe they can be exhaustive or use a simple strategy. But for a SQL DBMS, the search space is way too huge, and you need a way to navigate it intelligently. This "try the plan out" idea has come up many times, and it's a reasonable idea, but it's nothing new and not a complete solution. You need a way to eva…

I understand the issues you've brought up in your post as well as the traditional methods of pruning the search space of query plans through using left-deep join trees and histograms for calculating selectivity.

My top-level point was that there is no way humans will come up with the optimal query plan by hand, and like you said even if they do - the data will change.

As an aside, the reason Mongo can use such a simple but effective strategy is because it doesn't support joins, which makes calculating the optimal plan much, much easier. RethinkDB does, however, and as such, a CBO is that much more important.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#98

Post by pkolaczk »

btilly wrote: post #59
sixbrx wrote: post #48

Earlier quoted context omitted.

This accords with our experience with large Oracle databases as well. We've had a few panics caused by a DBA having updated table statistics when trying to optimize some query. This would occasionally cause sudden, massive changes in the way that other unrelated queries were performed, which queries would not finish and sometimes bring down the database. These experiences caused us to have to change our procedures an…

I wish that databases did "speculative" queries. In the simplest form, you could say, "I have 2 reasonable plans, let's try A, and if it takes above time X, then start B in parallel and go with whatever finishes first." You could ramp up the idea to handle changing query plans based on updated statistics by sending some fraction of queries to one plan, and some to another. Then keep stats on how that worked out for y…

DB/2 does that for ages.

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#99

Post by pkolaczk »

dennis82 wrote: post #40
josegonzalez wrote: post #35

Earlier quoted context omitted.

Which is good, because now your developers don't just add queries to the application without thinking about performance. You don't have to be a dba to understand how using one index vs another will affect the performance - and thus conversion rate - of your application.

no, it's categorically not good to make an application more fragile. A weakness restated is not a strength. Every database should have an optimizer, period.

If a database system does not support joins, aggregations or subqueries like most realtime NoSQL solutions do, an optimizer becomes pretty trivial. Optimizers are needed for analytical stuff. That's why most optimizers are evaluated on analytic workloads (e.g. TPC-H, TPC-W) not transactional / realtime (TPC-C).

Re: RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

#100

Post by troyk »

coffeemug wrote: post #73
troyk wrote: post #72

Am I the only one that hardly ever needs a case-sensitive query/index on string data? This is the one feature I miss from the MySQL glory days. With postgres, I use the citext type (case-insensitive text) but I guess Mongo and ReThinkDB expect you to either store the data as all lower or upper, or duplicate the field (for indexing).

The reason we (and probably Mongo folks) do it this way is because JSON is case-sensitive, so making indexes case-insensitive wouldn't work very well. EDIT: it looks like I misread your comment. I was talking about index/field names, while I think you were talking about field values. In this case Sam's comment below is correct -- once we add string.toLower() (which is very easy) you'll be able to do case-insensitive…

Installed and played with (1.3 I think) and it looks very promising but the ability to query and order by case-insensitive text is a requirement for us. Our app deals with user provided names of people, places and things that are commonly searched and sorted.

So not using RethinkDB today, but I am 100% in love with what you guys are trying to accomplish.

Post reply on HN