Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

T-Wand: beating Lucene in less than 600 lines of code

yyhh.org

Post reply on HN
Print view View original

81–90 of 96 posts

Re: T-Wand: beating Lucene in less than 600 lines of code

#81

Post by huahaiy »

softwaredoug wrote: post #80
huahaiy wrote: post #76

Earlier quoted context omitted.

You are making up straw man again. Did I make that claim? I was simply motivate my work, pointing out it's a problem that the current generation of search engine does not address. What is "relevance", it is of course a context sensitive question. You talk as if BM25 is the gold standard when it is not. The research on this is all over the place. I just read an article that says that BM25 is way worse than alternative…

The title of the section quoted is "Better Relevance" and I am simply saying, arguably, you can't make that claim without evidence. (That said, I do find much to like about the article and the WAND / T-WAND explanation. I'm only pointing out you can't claim 'better relevance' without a benchmark to go with it.)

You are mincing words again. It's a blog post we are talking about here.

In order to motive the work, I have to say what benefits it will bring. Agree?

It's my problem, and I get to define what is better relevance for me. In addition, this is also a well known problem acknowledge by Lucene documentation. So it is not an isolated problem. Some of the users voiced their agreement right here in comments. That's enough evidence for a blog post?

If you were writing this, how would you write it? You probably do worse.

I see that you work in Shopify Search, I hope that your attitude towards your users are not as hostile as this. It's the users who decide what are relevant to them, not you. I hope you remember that. Your manager will also probably agree.

Re: T-Wand: beating Lucene in less than 600 lines of code

#82

Post by antocv »

huahaiy wrote: post #64
ekzhu wrote: post #3

I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that: >> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of use…

It's your loss then. It is actually smug, to suggest the author, in this case, me, a computer scientist and a past professor who taught Information Retrieval class for more than 3 years, and who just came up with a new search algorithm, to "take a look at Wikipedia page for TF-IDF". In case you have not read the article due to your smugness, my search algorithm also uses TF-IDF and vector space model. Please do not d…

Haters gonna hate, dont mind them. Your work is truly amazing, I learned a lot and you opened a whole new world of datalog as an aside.

Re: T-Wand: beating Lucene in less than 600 lines of code

#83

Post by aleben »

huahaiy wrote: post #81
softwaredoug wrote: post #80

Earlier quoted context omitted.

The title of the section quoted is "Better Relevance" and I am simply saying, arguably, you can't make that claim without evidence. (That said, I do find much to like about the article and the WAND / T-WAND explanation. I'm only pointing out you can't claim 'better relevance' without a benchmark to go with it.)

You are mincing words again. It's a blog post we are talking about here. In order to motive the work, I have to say what benefits it will bring. Agree? It's my problem, and I get to define what is better relevance for me. In addition, this is also a well known problem acknowledge by Lucene documentation. So it is not an isolated problem. Some of the users voiced their agreement right here in comments. That's enough e…

Pardon my intrusion, but I think there's the necessity of toning down the discussion a bit. To be fair it is irrelevant where user A or user B works and I don't see any hostility in this discussion except the one from the blog's author. If the aim is doing Science, let's do this peacefully! I think the main problem that generated this discussion is the form of some sentences of the blog "As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that:

A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of user query terms. This makes perfect sense. Right?" This should have been changed in: "For the sake of this blog post I am interested in a simplified relevance concept: more query terms matched -> more relevance" In this way the blog would have not caused the outburst of (correct) indignation of the Information Retrieval community, that has been working for years on the topic, and knows very well how much complex is relevance.

Re: T-Wand: beating Lucene in less than 600 lines of code

#84

Post by softwaredoug »

aleben wrote: post #83
huahaiy wrote: post #81

Earlier quoted context omitted.

You are mincing words again. It's a blog post we are talking about here. In order to motive the work, I have to say what benefits it will bring. Agree? It's my problem, and I get to define what is better relevance for me. In addition, this is also a well known problem acknowledge by Lucene documentation. So it is not an isolated problem. Some of the users voiced their agreement right here in comments. That's enough e…

Pardon my intrusion, but I think there's the necessity of toning down the discussion a bit. To be fair it is irrelevant where user A or user B works and I don't see any hostility in this discussion except the one from the blog's author. If the aim is doing Science, let's do this peacefully! I think the main problem that generated this discussion is the form of some sentences of the blog "As someone who has a Ph.D. in…

Yes for sure! I agree. I mean no disrespect myself, the algorithm seems useful and I wish the author all the best. I know, from personal experience, having your work commented on can be stressful.

I'm just trying to understand what the author meant and their intention for anyone that might revisit this thread later. I think what you're saying sounds plausible as an interpretation.

Re: T-Wand: beating Lucene in less than 600 lines of code

#85

Post by throwaway744678 »

huahaiy wrote: post #78
juancb wrote: post #39

Earlier quoted context omitted.

It's fascinating seeing what hacker news thinks is smug. There are plenty of actually smug comments that go without being called out but for some reason this guy does. The PhD flex in jest by the blog post's author is a bit awkward but I don't know that I would characterize it as smug. Any insight into why the author is perceived as smug would be appreciated.

Let me offer my arguably biased insight: racism. I am obviously a Chinese. In the eyes of racists, Chinese Americans are supposed to be timid and does not make any noises, but I do make noises, so I am perceived as a smug today because I said I am a Ph.D, something else next time I said something else. It's not complicated.

Wow, I was with you until this. I, for one, never even suspected your origin or nationality reading the blog post.

OK, re-reading it, there are a few hints (a Chinese proverb, your nickname, etc.). But nothing "obvious".

Probably not the explanation we are looking for here.

Re: T-Wand: beating Lucene in less than 600 lines of code

#86

Post by fiedzia »

huahaiy wrote: post #70
diego wrote: post #4

I'm not sure if this is a troll. It seems the author did not spend much time learning Lucene, it already does everything he needs and better. Of course Lucene does tf-idf! If you are not getting the results you want, you must not be indexing right or must have something misconfigured. Lucene lets you tweak relevance, speed, index space efficiency, etc in a million ways.

I do not want to tweak Lucene. I don't want a huge dependency. I don't want Lucene. Period. I am writing a database, full-text search is just a *minor* feature. Why should I bringing in a huge dependency for a minor feature when my whole code base is orders magnitude smaller than that?

Do what you want, but do not claim "it's better than large dependency" when it does a tiny fraction of what Lucence does.

Re: T-Wand: beating Lucene in less than 600 lines of code

#87

Post by huahaiy »

aleben wrote: post #83
huahaiy wrote: post #81

Earlier quoted context omitted.

You are mincing words again. It's a blog post we are talking about here. In order to motive the work, I have to say what benefits it will bring. Agree? It's my problem, and I get to define what is better relevance for me. In addition, this is also a well known problem acknowledge by Lucene documentation. So it is not an isolated problem. Some of the users voiced their agreement right here in comments. That's enough e…

Pardon my intrusion, but I think there's the necessity of toning down the discussion a bit. To be fair it is irrelevant where user A or user B works and I don't see any hostility in this discussion except the one from the blog's author. If the aim is doing Science, let's do this peacefully! I think the main problem that generated this discussion is the form of some sentences of the blog "As someone who has a Ph.D. in…

That's the problem right there. The so called "Information Retrieval Community" does not seem to care about what users want, and they treat their users in a condescending way, ignoring an obvious problem for years, as clearly demonstrated here and elsewhere.

Why should I care about what they think? In the end of the day, they are serving the users. They are either going to be left behind, or they have to change.

For example, the newer search engines, such us, https://github.com/meilisearch/MeiliSearch, also do what T-Wand does. Users are going to vote by their feet. The "information retrieval community" be damned. It's as simple as that. Got it?

And change they will. What "information retrieval community" are you talking about? Just a bunch of fresh graduates who just got their PH.D, and some college students. That's about it. I don't see any IR professors coming out defending them.

Re: T-Wand: beating Lucene in less than 600 lines of code

#88

Post by huahaiy »

throwaway744678 wrote: post #85
huahaiy wrote: post #78

Earlier quoted context omitted.

Let me offer my arguably biased insight: racism. I am obviously a Chinese. In the eyes of racists, Chinese Americans are supposed to be timid and does not make any noises, but I do make noises, so I am perceived as a smug today because I said I am a Ph.D, something else next time I said something else. It's not complicated.

Wow, I was with you until this. I, for one, never even suspected your origin or nationality reading the blog post. OK, re-reading it, there are a few hints (a Chinese proverb, your nickname, etc.). But nothing "obvious". Probably not the explanation we are looking for here.

Not my nickname, my name.

It's good for you that you are not a racist (it's sad that such a normal thing has to be a compliment, but it is a compliment. Good for you.), but racists do look for these cues and make their judgement based on these.

It's an unfortunate current state of affairs that we have to live with. Ignoring it does not make it going away though. If you don't like it, you got to call it out whenever you see it, which I hope you do, instead of being taken back by it, as you seems to be.

Re: T-Wand: beating Lucene in less than 600 lines of code

#89

Post by huahaiy »

fiedzia wrote: post #86
huahaiy wrote: post #70

Earlier quoted context omitted.

I do not want to tweak Lucene. I don't want a huge dependency. I don't want Lucene. Period. I am writing a database, full-text search is just a *minor* feature. Why should I bringing in a huge dependency for a minor feature when my whole code base is orders magnitude smaller than that?

Do what you want, but do not claim "it's better than large dependency" when it does a tiny fraction of what Lucence does.

Did I claim that? Straw man much?

Re: T-Wand: beating Lucene in less than 600 lines of code

#90

Post by huahaiy »

antocv wrote: post #82
huahaiy wrote: post #64

Earlier quoted context omitted.

It's your loss then. It is actually smug, to suggest the author, in this case, me, a computer scientist and a past professor who taught Information Retrieval class for more than 3 years, and who just came up with a new search algorithm, to "take a look at Wikipedia page for TF-IDF". In case you have not read the article due to your smugness, my search algorithm also uses TF-IDF and vector space model. Please do not d…

Haters gonna hate, dont mind them. Your work is truly amazing, I learned a lot and you opened a whole new world of datalog as an aside.

Thank you. I am glad that you learned something from the article. I wrote the article for people like you, who are seeking knowledge and self improvement.
Post reply on HN