T-Wand: beating Lucene in less than 600 lines of code • Hacker News Board

Re: T-Wand: beating Lucene in less than 600 lines of code

#71

Post by huahaiy » Wed, Nov 17, 2021, 10:10 PM UTC

softwaredoug wrote: post #69

softwaredoug wrote: post #31

Earlier quoted context omitted.

As author of Relevant Search and contriburor of AI powered search I endorse this :) Relevance is really subjective, domain specific, requires intense amount of measurement and testing and many different ranking signals. Lucene is a toolbox for crafting many of these signals.

On second read-thru, I think the author is maybe(?) describing assumptions behind WAND and relevance algos that benefit from it? Maybe not some overarching statement about relevance per-se? But it's mixed in with statements about relevance / what Lucene does that are mostly incorrect... For example, he says > However, as you can see, this vector space model does not explicitly require a higher ranking document to con…

There's no guarantee in pure vector space model that it is the case. Your understanding is way off. This is a caveat that Lucene prominently put on their Web page.

https://lucene.apache.org/core/3_5_0/scoring.html

"Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on java-user@lucene.apache.org to figure out why a document with five of our query terms scores lower than a different document with only one of the query terms."

See? Lucene people know about it but they just do not think it is a problem.

But I do.

T-Wand, though also uses vector space model, makes sure it is the case.

Re: T-Wand: beating Lucene in less than 600 lines of code

#72

Post by huahaiy » Wed, Nov 17, 2021, 10:45 PM UTC

johncearls wrote: post #40

tgv wrote: post #30

Earlier quoted context omitted.

But a phd in HCI has very little relevance to information retrieval or computational linguistics.

Exactly this. I have a PHD in CS and am a world class expert in multi-omic data integration and analysis. I'm happy to throw my weight around in that area, but I'd never point to my PhD to pontificate on Neural nets or systems or queuing theory or 99 percent of CS. If getting a PhD doesn't teach you how much you don't know and how hard it is to develop real expertise in any area, I think you wasted your PhD. Note, my…

I think I am pretty qualified to make my declaration, since it is about user experience.

I also do know a lot about IR, because I taught Information Retrieval class for 3 years when I was teaching in university, I read research papers, and I just come up with a new search algorithm.

It's just that some people cannot accept that there are people who can cross fields with ease, make contributions quickly, and move on to the next field that pit their interests.

Yes, I am one of those people. In addition to HCI, I also published in the following areas: VR, DB, NLP, IR and Psychology. Sorry to hurt your feelings, but it is what it is. Accept it and move on.

Re: T-Wand: beating Lucene in less than 600 lines of code

#73

Post by softwaredoug » Wed, Nov 17, 2021, 10:47 PM UTC

huahaiy wrote: post #71

softwaredoug wrote: post #69

Earlier quoted context omitted.

On second read-thru, I think the author is maybe(?) describing assumptions behind WAND and relevance algos that benefit from it? Maybe not some overarching statement about relevance per-se? But it's mixed in with statements about relevance / what Lucene does that are mostly incorrect... For example, he says > However, as you can see, this vector space model does not explicitly require a higher ranking document to con…

There's no guarantee in pure vector space model that it is the case. Your understanding is way off. This is a caveat that Lucene prominently put on their Web page. https://lucene.apache.org/core/3_5_0/scoring.html "Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or do…

No there’s no guarantee. A very strong bias, but no absolute guarantee. Most people use AND queries by default or set a high enough min should match.

Those settings aside, arguably there are cases where fewer term matches could be more relevant.

For the search “Luke skywalker”

A tweet for example mentioning the term “Skywalker” once has much higher “Star Wars” aboutness than a move that uses skywalker one page and Luke pages apart.

That is the information density of Star Wars type content is far higher in the tweet, whereas eventually a book ends up using most English somewhere.

Re: T-Wand: beating Lucene in less than 600 lines of code

#74

Post by huahaiy » Wed, Nov 17, 2021, 10:55 PM UTC

softwaredoug wrote: post #73

huahaiy wrote: post #71

Earlier quoted context omitted.

There's no guarantee in pure vector space model that it is the case. Your understanding is way off. This is a caveat that Lucene prominently put on their Web page. https://lucene.apache.org/core/3_5_0/scoring.html "Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or do…

No there’s no guarantee. A very strong bias, but no absolute guarantee. Most people use AND queries by default or set a high enough min should match. Those settings aside, arguably there are cases where fewer term matches could be more relevant. For the search “Luke skywalker” A tweet for example mentioning the term “Skywalker” once has much higher “Star Wars” aboutness than a move that uses skywalker one page and Lu…

You either has or has no guarantee, that's what the word "guarantee" means. Don't mince words.

You had a wrong understanding. Now you are corrected. Let's move on.

Nobody said anything about "fewer term matches could not be more relevant". You are just making up straw men here. That's not the discussion we are having.

What I said is this, from a user point of view, it's not good to have a document containing fewer query terms to rank higher. This is a fact that even Lucene acknowledge (at least when they were version 3.5.0.). You have nothing to counter this fact.

Re: T-Wand: beating Lucene in less than 600 lines of code

#75

Post by softwaredoug » Wed, Nov 17, 2021, 11:08 PM UTC

huahaiy wrote: post #74

softwaredoug wrote: post #73

Earlier quoted context omitted.

No there’s no guarantee. A very strong bias, but no absolute guarantee. Most people use AND queries by default or set a high enough min should match. Those settings aside, arguably there are cases where fewer term matches could be more relevant. For the search “Luke skywalker” A tweet for example mentioning the term “Skywalker” once has much higher “Star Wars” aboutness than a move that uses skywalker one page and Lu…

You either has or has no guarantee, that's what the word "guarantee" means. Don't mince words. You had a wrong understanding. Now you are corrected. Let's move on. Nobody said anything about "fewer term matches could not be more relevant". You are just making up straw men here. That's not the discussion we are having. What I said is this, from a user point of view, it's not good to have a document containing fewer qu…

But you’re arguing that such a guarantee ALWAYS is most relevant. When that’s not always the case.

There’s been extensive research justification behind the vector-space model. BM25 is the 25th iteration of a model and well tuned BM25 holds the highest non nueral performance on many tasks including question answering[1]. Research has long found including factors other than total term matches matters. Such as IDF[2] and field length[3].

Have you benchmarked your relevance assumptions similarly? If so I’d love to see them and learn more!

1 - https://www.elastic.co/blog/improving-search-relevance-with-...

2- https://www.researchgate.net/publication/238123710_Understan...

3 - http://sifaka.cs.uiuc.edu/course/410s12/mir.pdf

Re: T-Wand: beating Lucene in less than 600 lines of code

#76

Post by huahaiy » Wed, Nov 17, 2021, 11:31 PM UTC

softwaredoug wrote: post #75

huahaiy wrote: post #74

Earlier quoted context omitted.

You either has or has no guarantee, that's what the word "guarantee" means. Don't mince words. You had a wrong understanding. Now you are corrected. Let's move on. Nobody said anything about "fewer term matches could not be more relevant". You are just making up straw men here. That's not the discussion we are having. What I said is this, from a user point of view, it's not good to have a document containing fewer qu…

But you’re arguing that such a guarantee ALWAYS is most relevant. When that’s not always the case. There’s been extensive research justification behind the vector-space model. BM25 is the 25th iteration of a model and well tuned BM25 holds the highest non nueral performance on many tasks including question answering[1]. Research has long found including factors other than total term matches matters. Such as IDF[2] an…

You are making up straw man again. Did I make that claim?

I was simply motivate my work, pointing out it's a problem that the current generation of search engine does not address.

What is "relevance", it is of course a context sensitive question.

You talk as if BM25 is the gold standard when it is not.

The research on this is all over the place. I just read an article that says that BM25 is way worse than alternative language models. You don't have to look far, for example this one that talks about Wand:

https://dl.acm.org/doi/10.1145/2537734.2537744

You know why I quit academia? It is useless arguments and virtual signalings like these.

I'd rather go out and build a damn thing that people like to use.

I merely pointed out that there's a problem, and I have a solution. I did not claim that my problem and my solution solve all problems. Isn't this obvious?

In my problem, my solution beats Lucene. It's as simple as that.

So if Lucene wants to be this infinitely configurable search library, it would be advisable to offer my solution as an option, or offer an better one that does something similar.

So far I have not seen any takers, only excuses.

Re: T-Wand: beating Lucene in less than 600 lines of code

#77

Post by [deleted] » Wed, Nov 17, 2021, 11:34 PM UTC

softwaredoug wrote: post #75

huahaiy wrote: post #74

Earlier quoted context omitted.

You either has or has no guarantee, that's what the word "guarantee" means. Don't mince words. You had a wrong understanding. Now you are corrected. Let's move on. Nobody said anything about "fewer term matches could not be more relevant". You are just making up straw men here. That's not the discussion we are having. What I said is this, from a user point of view, it's not good to have a document containing fewer qu…

But you’re arguing that such a guarantee ALWAYS is most relevant. When that’s not always the case. There’s been extensive research justification behind the vector-space model. BM25 is the 25th iteration of a model and well tuned BM25 holds the highest non nueral performance on many tasks including question answering[1]. Research has long found including factors other than total term matches matters. Such as IDF[2] an…

[deleted]

Re: T-Wand: beating Lucene in less than 600 lines of code

#78

Post by huahaiy » Thu, Nov 18, 2021, 12:18 AM UTC

juancb wrote: post #39

Cthulhu_ wrote: post #35

Earlier quoted context omitted.

Well yeah, but it shouldn't make you smug. A Ph.D, while not the beginning, is definitely not the end, and the people we look up to as Really Smart never sat on their laurels and quit learning, never considered themselves above others.

It's fascinating seeing what hacker news thinks is smug. There are plenty of actually smug comments that go without being called out but for some reason this guy does. The PhD flex in jest by the blog post's author is a bit awkward but I don't know that I would characterize it as smug. Any insight into why the author is perceived as smug would be appreciated.

Let me offer my arguably biased insight: racism.

I am obviously a Chinese.

In the eyes of racists, Chinese Americans are supposed to be timid and does not make any noises, but I do make noises, so I am perceived as a smug today because I said I am a Ph.D, something else next time I said something else.

It's not complicated.

Re: T-Wand: beating Lucene in less than 600 lines of code

#79

Post by huahaiy » Thu, Nov 18, 2021, 12:47 AM UTC

Cavallium wrote: post #67

huahaiy wrote: post #61

Earlier quoted context omitted.

Incorrect. The benchmark is fine. Your so-called "fundamental" flaws are just superficial things, such as using a different time measure, using a different benchmark library, passing in a thread pool, and so on. These are immaterial for relative comparison, unless proven otherwise. I have repeated asked you to send the code that does things properly in your mind, and you refused to do so. So you are just trolling. Pl…

I already explained to you why I didn't open more than one PR to the project, I'm not trolling anyone: without knowing Clojure the only contribution that I could do without spending literally days of work is just the use of System.nanoTime() instead of System.currentTimeMillis(). The reason why I don't write the other benchmarks in Java is because reimplementing anything from zero costs precious time, that I don't ha…

As I have repeated suggested, since you obviously know Lucene, it does not take much for you to write a few lines of Java code to say, "here, this is how it is supposed to be done".

Instead, you keeps giving all kind of excuses. You said you do not have time, but somehow you have time to write long wall of text like this. If you are not trolling, what you are doing then?

I pointed out to you why your suggestions will not make differences, some I have already tried. For example, initializing a query parser only once, but that would crash Lucene. Clearly, you do not know about this, so your knowledge of Lucene is not as good as you think you know.

Fair?

Re: T-Wand: beating Lucene in less than 600 lines of code

#80

Post by softwaredoug » Thu, Nov 18, 2021, 2:11 AM UTC

huahaiy wrote: post #76

softwaredoug wrote: post #75

Earlier quoted context omitted.

But you’re arguing that such a guarantee ALWAYS is most relevant. When that’s not always the case. There’s been extensive research justification behind the vector-space model. BM25 is the 25th iteration of a model and well tuned BM25 holds the highest non nueral performance on many tasks including question answering[1]. Research has long found including factors other than total term matches matters. Such as IDF[2] an…

You are making up straw man again. Did I make that claim? I was simply motivate my work, pointing out it's a problem that the current generation of search engine does not address. What is "relevance", it is of course a context sensitive question. You talk as if BM25 is the gold standard when it is not. The research on this is all over the place. I just read an article that says that BM25 is way worse than alternative…

The title of the section quoted is

"Better Relevance"

and I am simply saying, arguably, you can't make that claim without evidence.

(That said, I do find much to like about the article and the WAND / T-WAND explanation. I'm only pointing out you can't claim 'better relevance' without a benchmark to go with it.)