Live data from Hacker News

T-Wand: beating Lucene in less than 600 lines of code

yyhh.org

91–96 of 96 posts

Re: T-Wand: beating Lucene in less than 600 lines of code

#91
For what its worth, a final advise for those fresh graduates and college students. Don't take too seriously what your learned from books in school. They are not the gold standard the society measures things by. You got the wrong impression if you think your professors taught you that. They knew better. Don't take too much pride in your degree or in your specializations. They are nothing.

If you do not learn this now, your managers who maybe only has an associate degree from a community college, or from a school you look down on, will make you learn that soon enough.

These are not important things. An important thing is to keep an open mind and keep learning, so you are continuously employable. You can congratulate yourself when you are as old as me and can still do new things.

Re: T-Wand: beating Lucene in less than 600 lines of code

#92
post #78

Earlier quoted context omitted.

Let me offer my arguably biased insight: racism. I am obviously a Chinese. In the eyes of racists, Chinese Americans are supposed to be timid and does not make any noises, but I do make noises, so I am perceived as a smug today because I said I am a Ph.D, something else next time I said something else. It's not complicated.

Wow, I was with you until this. I, for one, never even suspected your origin or nationality reading the blog post. OK, re-reading it, there are a few hints (a Chinese proverb, your nickname, etc.). But nothing "obvious". Probably not the explanation we are looking for here.

BTW, you were with me? Where?

Why do you have to use a throwaway account to say this, I am curious.

Re: T-Wand: beating Lucene in less than 600 lines of code

#93
post #72

Earlier quoted context omitted.

Exactly this. I have a PHD in CS and am a world class expert in multi-omic data integration and analysis. I'm happy to throw my weight around in that area, but I'd never point to my PhD to pontificate on Neural nets or systems or queuing theory or 99 percent of CS. If getting a PhD doesn't teach you how much you don't know and how hard it is to develop real expertise in any area, I think you wasted your PhD. Note, my…

I think I am pretty qualified to make my declaration, since it is about user experience. I also do know a lot about IR, because I taught Information Retrieval class for 3 years when I was teaching in university, I read research papers, and I just come up with a new search algorithm. It's just that some people cannot accept that there are people who can cross fields with ease, make contributions quickly, and move on t…

That's not a new search algorithm. It's about the second one people have come up with. First version: all documents containing all search terms. Second version: counting frequency in document. Since you taught IR for three years, I'm surprised there's no mention of other relevant heuristics, or measuring against benchmarks.

Re: T-Wand: beating Lucene in less than 600 lines of code

#94
post #93
post #72

Earlier quoted context omitted.

I think I am pretty qualified to make my declaration, since it is about user experience. I also do know a lot about IR, because I taught Information Retrieval class for 3 years when I was teaching in university, I read research papers, and I just come up with a new search algorithm. It's just that some people cannot accept that there are people who can cross fields with ease, make contributions quickly, and move on t…

That's not a new search algorithm. It's about the second one people have come up with. First version: all documents containing all search terms. Second version: counting frequency in document. Since you taught IR for three years, I'm surprised there's no mention of other relevant heuristics, or measuring against benchmarks.

Again, you miss the novelty of the algorithm due to your smugness. I have updated the article to point out explicitly where the novelty is at. You completely ignored that. In fact, I believe you didn't even read that far before your jumping out to write this dismissal.

I wrote a blog post for lay people to read. I am under no obligation to do an extensive literature review, I only need to mention the essentials.

You just cannot seem to swallow the fact that someone who does not have a Ph.D in IR or computational linguistics can come in to contribute quickly then move on to something else. It hurts your feelings deeply, does it? Heed my advice, just get over it, don't take your degree or your specialization too seriously. It does not matter.

Re: T-Wand: beating Lucene in less than 600 lines of code

#95
One observation I have to make: this community of information retrieval seems to be very thin-faced and cannot seem to accept outside contributions. I hope I am wrong.

This is not an isolated case. They rejected some industry changing contributions when they first came out.

For example, I remember that Google PageRank paper was outright rejected by SiGIR. Lucky for us, that paper was rejected, shattering the dream of an easy academia career for the two Google founders. Otherwise, we would probably still be searching with hotbots (remember that?).

The WAND paper was published in CIKM, a lowly conference in some people's eyes, I guess that's not the first venue they tried to publish that work in, because I worked at IBM Research before, I knew for a fact that CIKM was not in the first tier of the venues to publish, definitely not for the IR PIC.

Very interesting. I wonder what have caused this. My wild guess is that this may has something to do with early mistreatment the pioneering IR people received from the wider CS community, because their field was probably considered "soft" and not hard enough. For instance, even today, IR papers are not full of theorems and proofs, unlike some other CS fields, such as database. Their revengeful attitude got passed down to their students, as we can see here. And their reasons for dismissing other people's work are all the same: this is not new, we have done this before. But you have not, because the proof is in the systems you build and the papers you publish.

It does not have to be this way. HCI is even softer, but HCI is the most open field in CS, anyone can publish, you can prove theorems, or you can present user studies, anything goes. Funding? Not a problem. Because it is so open, HCI can work with anyone and on any projects. I remember that my advisor had so much funding, he basically paid me to whatever I wanted when I was in school. I am deeply in gratitude for that openness.

Re: T-Wand: beating Lucene in less than 600 lines of code

#96
post #83
post #81

Earlier quoted context omitted.

You are mincing words again. It's a blog post we are talking about here. In order to motive the work, I have to say what benefits it will bring. Agree? It's my problem, and I get to define what is better relevance for me. In addition, this is also a well known problem acknowledge by Lucene documentation. So it is not an isolated problem. Some of the users voiced their agreement right here in comments. That's enough e…

Pardon my intrusion, but I think there's the necessity of toning down the discussion a bit. To be fair it is irrelevant where user A or user B works and I don't see any hostility in this discussion except the one from the blog's author. If the aim is doing Science, let's do this peacefully! I think the main problem that generated this discussion is the form of some sentences of the blog "As someone who has a Ph.D. in…

The "hostility" I referred to was the one towards the users. Basically, he's demanding users to show evidence that whatever they want is better than BM25.

That's just wrong. Period. This is the reason I mention his job, because his job requires him to listen to the users, instead of taking something he learned in school and use that to talk condescendingly to his users. That's not gonna fly.

Given this context, I think whatever I said it is entirely appropriate.

Post reply on HN