Related, I recently published https://github.com/gajus/liqe, although not 600 lines of code
T-Wand: beating Lucene in less than 600 lines of code
21–30 of 96 posts
Re: T-Wand: beating Lucene in less than 600 lines of code
#22Given I'm quite a newb at this field - found article interesting despite having (probably deserved) criticism.
The critics are all Lucene developers or deeply invested in that. The criticism boils down to well "Simply just know Lucene and handle the million lines of code by writing plugins and configurations to get the same result as OP", except with worse performance.
Re: T-Wand: beating Lucene in less than 600 lines of code
#23Earlier quoted context omitted.
Perhaps the OP just wanted a simple custom algorithm that would be easy to integrate? I do not understand why rejecting a million-lines-of-code dependency implies the person is a troll?
I think you are almost be definition trolling when you write beating X (millions lines of code) with 600 lines. because when you say 'beating' the implication is it does what X does better - as opposed to just matching my use case with 600 lines and not big dependency import. this usage of beating is going to rile up (i.e troll) everyone familiar with the subject who can see thousands of ways you did not beat it.
The funny thing is that the benchmarks that he wrote are more problematic than the T-WAND code itself.
He didn't use any benchmarking library, and he used System.currentTimeMillis instead of the high precision timer available with System.nanoTime.
He also instantiated the IndexSearcher without specifying any executor and he instantiated the legacy query parser on every lucene search
Re: T-Wand: beating Lucene in less than 600 lines of code
#24Earlier quoted context omitted.
I think you are almost be definition trolling when you write beating X (millions lines of code) with 600 lines. because when you say 'beating' the implication is it does what X does better - as opposed to just matching my use case with 600 lines and not big dependency import. this usage of beating is going to rile up (i.e troll) everyone familiar with the subject who can see thousands of ways you did not beat it.
I think the heading is fine. They are beating Lucene in the context of their particular problem. Any additional features that a product has but which you don’t need for a particular purpose is a downside, not an upside. They’re not claiming to beat Lucene for the general case of everything Lucene is capable of.
He is beating lucene only if you look at his broken benchmark
Re: T-Wand: beating Lucene in less than 600 lines of code
#25I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that: >> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of use…
I also suggest you to look at his "benchmarks" code
Re: T-Wand: beating Lucene in less than 600 lines of code
#26I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that: >> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of use…
Yeah beyond the cringe of thinking a Ph.D really means anything, its just highlights the pure lack of Lucene knowledge.
A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.
Re: T-Wand: beating Lucene in less than 600 lines of code
#27So if someone created a document consisting of just these search terms (and nothing else) in the search order, that's always going to be the highest ranked item, right, despite a lack of any added information, right?
Forgive me if this is a dumb question, Clojure isn't a language I know at all so it was a little hard to parse the code, I couldn't figure out if there was something that indicated a minimum levenshtein distance threshold for documents, but I feel like only looking at documents which are longer than the search query would also be a reasonable optimization.
Re: T-Wand: beating Lucene in less than 600 lines of code
#28Given I'm quite a newb at this field - found article interesting despite having (probably deserved) criticism.
The critics are all Lucene developers or deeply invested in that. The criticism boils down to well "Simply just know Lucene and handle the million lines of code by writing plugins and configurations to get the same result as OP", except with worse performance.
That's not true. The real problem is that he is implying that his software is faster than Lucene showing data from a benchmark that has substantial flaws.
The reason why the majority of criticism comes from Lucene users is because they generally have a higher knownledge about this field than the people that read this article without having the basic knowledge to form any kind of criticism
Re: T-Wand: beating Lucene in less than 600 lines of code
#29As a user, when I write a search query as a space-separated list of n words, I expect to see documents which contain all of those words (or synonyms). I absolutely don't want to see documents which contain only n-1 of the specified words. I know exactly what I want, and the reason I've added the last word to the query is that there were too many generic, boring, irrelevant results without it.
Re: T-Wand: beating Lucene in less than 600 lines of code
#30Earlier quoted context omitted.
Yeah beyond the cringe of thinking a Ph.D really means anything, its just highlights the pure lack of Lucene knowledge.
A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.
But a phd in HCI has very little relevance to information retrieval or computational linguistics.