Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

T-Wand: beating Lucene in less than 600 lines of code

yyhh.org

Post reply on HN
Print view View original

51–60 of 96 posts

Re: T-Wand: beating Lucene in less than 600 lines of code

#51

Post by huahaiy »

Cavallium wrote: post #16

The benchmarks in question have several implementation issues, I reported them on GitHub. https://github.com/juji-io/datalevin/issues/created_by/caval...

I would appreciate if you also send a PR to address these issues, since clearly you know more about benchmarking than me.

Contributing to open source project is good, right?

BTW, Lucene is an open source project, Datalevin is also an open source project. Contributing to either would be equally good.

Even if you only want to contribute to Lucene for some reasons, a better contribution would be to integrate T-Wand in Lucene, rather than trying to talk down Datalevin. Would you agree?

Re: T-Wand: beating Lucene in less than 600 lines of code

#52

Post by huahaiy »

Cavallium wrote: post #49
huahaiy wrote: post #44

I would appreciate people who claims that "the author does not know Lucene enough" to send a Pull Request, where Lucene is configured to do the same thing that T-Wand does, that is to guarantee documents containing more query terms to rank higher than those containing less query terms. To those who says "I like Lucene because it does Tf-idf", please be assured that T-Wand also does tf-idf and uses vector space model.…

I opened a PR to use the more precise timers. This would not improve the performance of any of the tests, but it will improve their time accuracy. I highlighted some other problems related to the Lucene benchmark, but since I can't program in Clojure I can't fix them with a PR: I opened 3 issues to describe what can be done to address them.

Merged your PR. Thank you! If you do not know Clojure, that's fine, just outline what you want to do in Java, I can translate that into Clojure. Thanks.

To be honest, all your proposed changes would not make material difference, for algorithmic differences cannot be made up by implementation details.

As has already been demonstrated, T-Wand is implemented in a much slower language, Clojure. Clojure is implemented on top of JVM, it cannot possibly beat the same algorithm written in optimized Java. The only way to make a difference is to change the algorithm. So all your suggestions would not make a difference.

Re: T-Wand: beating Lucene in less than 600 lines of code

#53

Post by huahaiy »

diego wrote: post #7
twhitmore wrote: post #5

Interesting. I can't say Lucene gives great results for relevance on a large database, eg. Confluence. However there is a large space between algorithmic assumptions and what "relevance" actually is to the user. This article may improve, but my feeling is -- both for search and for the article -- that more context is needed.

Lucene is extremely flexible. You have to configure it to do what you want if you don't like the defaults (which OP didn't do).

Show me how to do that? I will integrate that into the benchmark. Thanks.

Re: T-Wand: beating Lucene in less than 600 lines of code

#54

Post by gnabgib »

huahaiy wrote: post #44

I would appreciate people who claims that "the author does not know Lucene enough" to send a Pull Request, where Lucene is configured to do the same thing that T-Wand does, that is to guarantee documents containing more query terms to rank higher than those containing less query terms. To those who says "I like Lucene because it does Tf-idf", please be assured that T-Wand also does tf-idf and uses vector space model.…

Why are you referring to yourself (the author) in the third person? It seems.. disingenuous

Re: T-Wand: beating Lucene in less than 600 lines of code

#55

Post by [deleted] »

gnabgib wrote: post #54
huahaiy wrote: post #44

I would appreciate people who claims that "the author does not know Lucene enough" to send a Pull Request, where Lucene is configured to do the same thing that T-Wand does, that is to guarantee documents containing more query terms to rank higher than those containing less query terms. To those who says "I like Lucene because it does Tf-idf", please be assured that T-Wand also does tf-idf and uses vector space model.…

Why are you referring to yourself (the author) in the third person? It seems.. disingenuous

[deleted]

Re: T-Wand: beating Lucene in less than 600 lines of code

#56

Post by Cavallium »

huahaiy wrote: post #51
Cavallium wrote: post #16

The benchmarks in question have several implementation issues, I reported them on GitHub. https://github.com/juji-io/datalevin/issues/created_by/caval...

I would appreciate if you also send a PR to address these issues, since clearly you know more about benchmarking than me. Contributing to open source project is good, right? BTW, Lucene is an open source project, Datalevin is also an open source project. Contributing to either would be equally good. Even if you only want to contribute to Lucene for some reasons, a better contribution would be to integrate T-Wand in L…

I never had the interest to "talk down Datalevin", I criticized the results because the title was quite misleading in the tones and the benchmarks were not rigorous enough to express conclusive statements.

Re: T-Wand: beating Lucene in less than 600 lines of code

#57

Post by huahaiy »

gnabgib wrote: post #54
huahaiy wrote: post #44

I would appreciate people who claims that "the author does not know Lucene enough" to send a Pull Request, where Lucene is configured to do the same thing that T-Wand does, that is to guarantee documents containing more query terms to rank higher than those containing less query terms. To those who says "I like Lucene because it does Tf-idf", please be assured that T-Wand also does tf-idf and uses vector space model.…

Why are you referring to yourself (the author) in the third person? It seems.. disingenuous

Corrected. Thanks. I am not a native English speaker, so please do not read too much into it.

Re: T-Wand: beating Lucene in less than 600 lines of code

#58

Post by huahaiy »

Cavallium wrote: post #56
huahaiy wrote: post #51

Earlier quoted context omitted.

I would appreciate if you also send a PR to address these issues, since clearly you know more about benchmarking than me. Contributing to open source project is good, right? BTW, Lucene is an open source project, Datalevin is also an open source project. Contributing to either would be equally good. Even if you only want to contribute to Lucene for some reasons, a better contribution would be to integrate T-Wand in L…

I never had the interest to "talk down Datalevin", I criticized the results because the title was quite misleading in the tones and the benchmarks were not rigorous enough to express conclusive statements.

It's a personal blog post. If the title is not "misleading", which it is not, you would not see it, would you?

Re: T-Wand: beating Lucene in less than 600 lines of code

#59

Post by huahaiy »

The_rationalist wrote: post #38
teraflop wrote: post #11

Earlier quoted context omitted.

Technically the author's description of how Lucene's ranking works is accurate. But the catch is that Lucene's default "vector space model" treats each document as a very high-dimensional vector, where the components are the words in the document (scaled using tf-idf). Under this definition, "similarity" reduces to a normalized dot-product of the query vector and the document vector. So in practice, it does roughly w…

isn't BM25 used by default instead of tf-idf?

BM25 is just a variant of tf-idf.

Re: T-Wand: beating Lucene in less than 600 lines of code

#60

Post by huahaiy »

tgv wrote: post #30
speedgoose wrote: post #26

Earlier quoted context omitted.

A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.

But a phd in HCI has very little relevance to information retrieval or computational linguistics.

It does have relevance to user experience, which is under discussion.
Post reply on HN