Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

T-Wand: beating Lucene in less than 600 lines of code

yyhh.org

Post reply on HN
Print view View original

41–50 of 96 posts

Re: T-Wand: beating Lucene in less than 600 lines of code

#41

Post by juancb »

tgv wrote: post #30
speedgoose wrote: post #26

Earlier quoted context omitted.

A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.

But a phd in HCI has very little relevance to information retrieval or computational linguistics.

Of course it doesn't and the author doesn't try to make that claim. The author jokes that their HCI PhD is only good enough to give them the authority to make obvious statementd about what a "good" search experience should be for a user.

What did I miss?

Re: T-Wand: beating Lucene in less than 600 lines of code

#43

Post by taeric »

juancb wrote: post #39
Cthulhu_ wrote: post #35

Earlier quoted context omitted.

Well yeah, but it shouldn't make you smug. A Ph.D, while not the beginning, is definitely not the end, and the people we look up to as Really Smart never sat on their laurels and quit learning, never considered themselves above others.

It's fascinating seeing what hacker news thinks is smug. There are plenty of actually smug comments that go without being called out but for some reason this guy does. The PhD flex in jest by the blog post's author is a bit awkward but I don't know that I would characterize it as smug. Any insight into why the author is perceived as smug would be appreciated.

Agreed. The wink right after it, I would have thought, dulled any smugness.

I could see calling it audacious. But, our industry advances on audacity.

Re: T-Wand: beating Lucene in less than 600 lines of code

#44

Post by huahaiy »

I would appreciate people who claims that "the author does not know Lucene enough" to send a Pull Request, where Lucene is configured to do the same thing that T-Wand does, that is to guarantee documents containing more query terms to rank higher than those containing less query terms.

To those who says "I like Lucene because it does Tf-idf", please be assured that T-Wand also does tf-idf and uses vector space model.

To those who say "T-Wand is just Wand", please be reminded that you are just repeating what the article was saying. Of course T-Wand is Wand, but you miss the technical contribution of T-Wand, which is clearly laid out in "What's new in T-Wand" section of the article. Of course, it is understandable if you are not able to judge the merit of that contribution, for that's the job of the peer review panel, which I do not expect Hacker News readers to be. This work will be submitted to a peer reviewed conference with more experiments added and all trade-offs discussed.

This may come as a disappointment to some, but the paper will not have a comparison with Lucene, for it would be scientifically unsound.

However, it should be easy for Lucene to add T-Wand as an option, and I do hope so.

Re: T-Wand: beating Lucene in less than 600 lines of code

#45

Post by huahaiy »

fizx wrote: post #6

Lucene has the concept of "minimum match," [1] which is what you're playing with. You can implement your algorithm on top of this relatively trivially, by re-querying with min_match=n, min_match=n-1, etc until you get results. If you wrote a Lucene plugin to do this, it would also be small ( People have chosen not to do this because (1) it adds complexity, and (2) the bold declaration you made (A good top-K algorithm…

The industry has not "chosen" to not do what T-Wand does, the industry did not have that option because T-Wand just comes out. Now with T-Wand, the industry has that choice to make.

It is easy to implement T-Wand in the existing Lucene code base, and I would hope so. Otherwise, it would be a shame.

Re: T-Wand: beating Lucene in less than 600 lines of code

#46

Post by huahaiy »

Cavallium wrote: post #24
codetrotter wrote: post #15

Earlier quoted context omitted.

I think the heading is fine. They are beating Lucene in the context of their particular problem. Any additional features that a product has but which you don’t need for a particular purpose is a downside, not an upside. They’re not claiming to beat Lucene for the general case of everything Lucene is capable of.

He is beating lucene only if you look at his broken benchmark

Send a PR if you feel the benchmark can be improved. I will be happy to merge it and rerun the benchmark.

Re: T-Wand: beating Lucene in less than 600 lines of code

#47

Post by [deleted] »

Cavallium wrote: post #23
bryanrasmussen wrote: post #13

Earlier quoted context omitted.

I think you are almost be definition trolling when you write beating X (millions lines of code) with 600 lines. because when you say 'beating' the implication is it does what X does better - as opposed to just matching my use case with 600 lines and not big dependency import. this usage of beating is going to rile up (i.e troll) everyone familiar with the subject who can see thousands of ways you did not beat it.

The funny thing is that the benchmarks that he wrote are more problematic than the T-WAND code itself. He didn't use any benchmarking library, and he used System.currentTimeMillis instead of the high precision timer available with System.nanoTime. He also instantiated the IndexSearcher without specifying any executor and he instantiated the legacy query parser on every lucene search

[deleted]

Re: T-Wand: beating Lucene in less than 600 lines of code

#48

Post by huahaiy »

lsiebert wrote: post #27

So if someone created a document consisting of just these search terms (and nothing else) in the search order, that's always going to be the highest ranked item, right, despite a lack of any added information, right? Forgive me if this is a dumb question, Clojure isn't a language I know at all so it was a little hard to parse the code, I couldn't figure out if there was something that indicated a minimum levenshtein…

Correct. That is also what all search engines will currently do.

Re: T-Wand: beating Lucene in less than 600 lines of code

#49

Post by Cavallium »

huahaiy wrote: post #44

I would appreciate people who claims that "the author does not know Lucene enough" to send a Pull Request, where Lucene is configured to do the same thing that T-Wand does, that is to guarantee documents containing more query terms to rank higher than those containing less query terms. To those who says "I like Lucene because it does Tf-idf", please be assured that T-Wand also does tf-idf and uses vector space model.…

I opened a PR to use the more precise timers. This would not improve the performance of any of the tests, but it will improve their time accuracy.

I highlighted some other problems related to the Lucene benchmark, but since I can't program in Clojure I can't fix them with a PR: I opened 3 issues to describe what can be done to address them.

Re: T-Wand: beating Lucene in less than 600 lines of code

#50

Post by huahaiy »

diego wrote: post #12
grumpyprole wrote: post #10

Earlier quoted context omitted.

Perhaps the OP just wanted a simple custom algorithm that would be easy to integrate? I do not understand why rejecting a million-lines-of-code dependency implies the person is a troll?

I meant as in "beating Lucene" without trying to use Lucene properly. It's like trying to use a spoon to cut a steak and then saying that you can beat silverware with your teeth.

Send a PR with Lucene properly used. It's an open source project, and I said I welcome any contribution in the article. Please do.
Post reply on HN