Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

T-Wand: beating Lucene in less than 600 lines of code

yyhh.org

Post reply on HN
Print view View original

61–70 of 96 posts

Re: T-Wand: beating Lucene in less than 600 lines of code

#61

Post by huahaiy »

Cavallium wrote: post #28
antocv wrote: post #22

Earlier quoted context omitted.

The critics are all Lucene developers or deeply invested in that. The criticism boils down to well "Simply just know Lucene and handle the million lines of code by writing plugins and configurations to get the same result as OP", except with worse performance.

That's not true. The real problem is that he is implying that his software is faster than Lucene showing data from a benchmark that has substantial flaws. The reason why the majority of criticism comes from Lucene users is because they generally have a higher knownledge about this field than the people that read this article without having the basic knowledge to form any kind of criticism

Incorrect. The benchmark is fine. Your so-called "fundamental" flaws are just superficial things, such as using a different time measure, using a different benchmark library, passing in a thread pool, and so on. These are immaterial for relative comparison, unless proven otherwise.

I have repeated asked you to send the code that does things properly in your mind, and you refused to do so.

So you are just trolling. Please stop hinder the progress of the industry through pointless trolling. Makes some positive contributions instead, and I have pointed out repeatedly how to make a positive contribution in this case. Please do.

You are a university student, I am a seasoned computer scientist, past researcher and professor, I am busying running a startup. This will be the last time I will say this to you: please consider contributing something positive to the world.

Re: T-Wand: beating Lucene in less than 600 lines of code

#62

Post by huahaiy »

Cavallium wrote: post #25
ekzhu wrote: post #3

I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that: >> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of use…

I also suggest you to look at his "benchmarks" code

The benchmark is fine. Your so-called "fundamental" flaws are just superficial things, such as using a different time measure, using a different benchmark library, passing in a thread pool, and so on. These are immaterial to relative comparison, unless proven otherwise.

I have repeated asked you to send the code that does things properly in your mind, and you refused to do so.

So you are just trolling. Please stop hinder the progress of the industry through pointless trolling. Makes some positive contributions instead, and I have pointed out repeatedly how to make a positive contribution in this case. Please do.

You are a university student, I am a seasoned computer scientist, past researcher and professor, I am busying running a startup. This will be the last time I will say this to you: please consider contributing something positive to the world.

Re: T-Wand: beating Lucene in less than 600 lines of code

#63

Post by huahaiy »

ysleepy wrote: post #18

The tone of the article makes it incredibly annoying and the author 100% lands on the hall of shame list of ridiculous benchmarks he mentioned. tf-idf and phrase search alone make lucene 10x better, not speaking of the compression, fuzzy search, advanced query features etc. I love new search engines like tantivy etc, but this is, due to the tone, a sad joke.

It would really be a sad joke if one dismisses a good technology just because he does not like the tone of the author of the said technology.

Besides, the said technology does do tf-idf and fuzzy search, and will soon have phrase search, and other features, if enough people request them.

Datalevin is an open source project, not unlike Lucene. What's the difference? Not like the tone?

Re: T-Wand: beating Lucene in less than 600 lines of code

#64

Post by huahaiy »

ekzhu wrote: post #3

I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that: >> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of use…

It's your loss then.

It is actually smug, to suggest the author, in this case, me, a computer scientist and a past professor who taught Information Retrieval class for more than 3 years, and who just came up with a new search algorithm, to "take a look at Wikipedia page for TF-IDF".

In case you have not read the article due to your smugness, my search algorithm also uses TF-IDF and vector space model.

Please do not dis-understand what is going on here: I am running a startup, and I am also old enough to not care about publications as much as people who are younger or in academia.

That's why I chose to reveal this in a blog post instead of hiding it until after my paper is published. Understood?

Re: T-Wand: beating Lucene in less than 600 lines of code

#65

Post by huahaiy »

Cavallium wrote: post #32
ptspts wrote: post #29

As a user, when I write a search query as a space-separated list of n words, I expect to see documents which contain all of those words (or synonyms). I absolutely don't want to see documents which contain only n-1 of the specified words. I know exactly what I want, and the reason I've added the last word to the query is that there were too many generic, boring, irrelevant results without it.

You completely misunderstood what this project does. This kind of libraries do not force you to return results in a specific way. Generally they allow the programmer to customize how the results are scored and chosen.

I don't think he misunderstood anything.

There's the kind of library Lucene wants to be, a configurable library. But there are other libraries do not want that.

In this case, Datalevin is a database system, full-text search is just but a feature, I do not want users to configure full-text search in Datalevin as much as they like in Lucene. I want the default to be good, which Lucene's is not.

Re: T-Wand: beating Lucene in less than 600 lines of code

#66

Post by huahaiy »

taeric wrote: post #43
juancb wrote: post #39

Earlier quoted context omitted.

It's fascinating seeing what hacker news thinks is smug. There are plenty of actually smug comments that go without being called out but for some reason this guy does. The PhD flex in jest by the blog post's author is a bit awkward but I don't know that I would characterize it as smug. Any insight into why the author is perceived as smug would be appreciated.

Agreed. The wink right after it, I would have thought, dulled any smugness. I could see calling it audacious. But, our industry advances on audacity.

Right, I thought that the wink is enough, and I have also paid enough homage to Lucene.

What's wrong with people? I think a lot of them are probably just university students who have nothing better to do than to troll on Internet.

Re: T-Wand: beating Lucene in less than 600 lines of code

#67

Post by Cavallium »

huahaiy wrote: post #61
Cavallium wrote: post #28

Earlier quoted context omitted.

That's not true. The real problem is that he is implying that his software is faster than Lucene showing data from a benchmark that has substantial flaws. The reason why the majority of criticism comes from Lucene users is because they generally have a higher knownledge about this field than the people that read this article without having the basic knowledge to form any kind of criticism

Incorrect. The benchmark is fine. Your so-called "fundamental" flaws are just superficial things, such as using a different time measure, using a different benchmark library, passing in a thread pool, and so on. These are immaterial for relative comparison, unless proven otherwise. I have repeated asked you to send the code that does things properly in your mind, and you refused to do so. So you are just trolling. Pl…

I already explained to you why I didn't open more than one PR to the project, I'm not trolling anyone: without knowing Clojure the only contribution that I could do without spending literally days of work is just the use of System.nanoTime() instead of System.currentTimeMillis(). The reason why I don't write the other benchmarks in Java is because reimplementing anything from zero costs precious time, that I don't have. If you don't have time too, instead of closing the issues and calling me a troll, you can just say that; you are taking some valid critiques as a personal attack.

It's true that I'm just an university student and you are a scientist, a researcher, a professor, and an enterpreneur, but it doesn't mean that I can't be knowledged as you in very narrow fields. I'm currently running a personal project that uses Lucene since four years ago, with about 8 billion messages and 340 million chats stored in a distributed lucene index, so I'm not the regular troll or a person that just talks without knowing anything. I surely can't critique your T-WAND algorithm, that it seems to be good in your use case, but with all the humilty in the world I can say to have the minimum knowledge required to spot some weird usages of Lucene APIs.

Re: T-Wand: beating Lucene in less than 600 lines of code

#68

Post by huahaiy »

ot wrote: post #9

The author deeply misunderstands how WAND works, and "T-WAND" is just WAND. Once a pivot is found, you don't have to advance all the iterators that precede it, but just enough to disprove the current document, at which point a new pivot is chosen (this is equivalent to the heuristic described in the article). The order in which iterators are tested against the pivot can be term cardinality ordering (which is one of t…

Feeling smug?

You fundamentally misunderstand both Wand and T-Wand.

I suggest you to read my article and the papers again.

The novelty of T-Wand, is not in sorting the rows on idf, which T-Wand actually does NOT do, but in adding some filtering conditions that are not previously reported in the literature.

You missed that, so you missed the whole point. Smugness hurts.

I see that you work on Facebook Search and I know you are one of the authors in this field, I can see why your feelings are hurt.

It's okay. There are always people smarter than you are, who can come into a completely new field, spend a few weeks, and come up with something new that you could not come up with when you have spent your whole career on.

It's OK. When you are older, you will learn to accept that. You can only work with what life gives you.

Please take my ideas and put that in Faceook Search, because it presently sucks.

BTW, you should have put the PDF of your paper on the Web, then I would have read it, instead of only recognizing your name, put it on arxiv like everyone else, if you want people to read it.

Also, that's why I post it on hacker news instead of publishing first in a conference. Because there will be more people reading it. Not many people read papers.

Re: T-Wand: beating Lucene in less than 600 lines of code

#69

Post by softwaredoug »

softwaredoug wrote: post #31
ekzhu wrote: post #3

I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that: >> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of use…

As author of Relevant Search and contriburor of AI powered search I endorse this :) Relevance is really subjective, domain specific, requires intense amount of measurement and testing and many different ranking signals. Lucene is a toolbox for crafting many of these signals.

On second read-thru, I think the author is maybe(?) describing assumptions behind WAND and relevance algos that benefit from it? Maybe not some overarching statement about relevance per-se? But it's mixed in with statements about relevance / what Lucene does that are mostly incorrect...

For example, he says

> However, as you can see, this vector space model does not explicitly require a higher ranking document to contain more query terms than a lower ranking one.

Well the way you get a higher similarity in a vector-space model is matching more terms. The caveat being that IDF and field length makes you also consider a term's specificity. So if you search for 'luke skywalker' you care more about the 'skywalker' match than the 'luke' match. But a match on BOTH 'luke skywalker' would score higher (field lengths being constant)

Re: T-Wand: beating Lucene in less than 600 lines of code

#70

Post by huahaiy »

diego wrote: post #4

I'm not sure if this is a troll. It seems the author did not spend much time learning Lucene, it already does everything he needs and better. Of course Lucene does tf-idf! If you are not getting the results you want, you must not be indexing right or must have something misconfigured. Lucene lets you tweak relevance, speed, index space efficiency, etc in a million ways.

I do not want to tweak Lucene. I don't want a huge dependency. I don't want Lucene. Period.

I am writing a database, full-text search is just a *minor* feature.

Why should I bringing in a huge dependency for a minor feature when my whole code base is orders magnitude smaller than that?

Post reply on HN