Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

T-Wand: beating Lucene in less than 600 lines of code

yyhh.org

Post reply on HN
Print view View original

11–20 of 96 posts

Re: T-Wand: beating Lucene in less than 600 lines of code

#11

Post by teraflop »

jhgb wrote: post #8

> Most search engines use something called a vector space model, where both user queries and documents are reduced to vectors (i.e. a fixed number of numbers). That is to say, the search engines are not looking at the meanings of the queries or the documents. Instead, they turned them both into some numbers. The search problem, is reduced to a problem of finding the similarity between the numbers representing the que…

Technically the author's description of how Lucene's ranking works is accurate.

But the catch is that Lucene's default "vector space model" treats each document as a very high-dimensional vector, where the components are the words in the document (scaled using tf-idf). Under this definition, "similarity" reduces to a normalized dot-product of the query vector and the document vector. So in practice, it does roughly what you want: documents that match more query words end up getting more non-zero terms in their similarity score, which leads to a higher ranking.

Re: T-Wand: beating Lucene in less than 600 lines of code

#12

Post by diego »

grumpyprole wrote: post #10
diego wrote: post #4

I'm not sure if this is a troll. It seems the author did not spend much time learning Lucene, it already does everything he needs and better. Of course Lucene does tf-idf! If you are not getting the results you want, you must not be indexing right or must have something misconfigured. Lucene lets you tweak relevance, speed, index space efficiency, etc in a million ways.

Perhaps the OP just wanted a simple custom algorithm that would be easy to integrate? I do not understand why rejecting a million-lines-of-code dependency implies the person is a troll?

I meant as in "beating Lucene" without trying to use Lucene properly. It's like trying to use a spoon to cut a steak and then saying that you can beat silverware with your teeth.

Re: T-Wand: beating Lucene in less than 600 lines of code

#13

Post by bryanrasmussen »

grumpyprole wrote: post #10
diego wrote: post #4

I'm not sure if this is a troll. It seems the author did not spend much time learning Lucene, it already does everything he needs and better. Of course Lucene does tf-idf! If you are not getting the results you want, you must not be indexing right or must have something misconfigured. Lucene lets you tweak relevance, speed, index space efficiency, etc in a million ways.

Perhaps the OP just wanted a simple custom algorithm that would be easy to integrate? I do not understand why rejecting a million-lines-of-code dependency implies the person is a troll?

I think you are almost be definition trolling when you write beating X (millions lines of code) with 600 lines.

because when you say 'beating' the implication is it does what X does better - as opposed to just matching my use case with 600 lines and not big dependency import.

this usage of beating is going to rile up (i.e troll) everyone familiar with the subject who can see thousands of ways you did not beat it.

Re: T-Wand: beating Lucene in less than 600 lines of code

#14

Post by jhgb »

teraflop wrote: post #11
jhgb wrote: post #8

> Most search engines use something called a vector space model, where both user queries and documents are reduced to vectors (i.e. a fixed number of numbers). That is to say, the search engines are not looking at the meanings of the queries or the documents. Instead, they turned them both into some numbers. The search problem, is reduced to a problem of finding the similarity between the numbers representing the que…

Technically the author's description of how Lucene's ranking works is accurate. But the catch is that Lucene's default "vector space model" treats each document as a very high-dimensional vector, where the components are the words in the document (scaled using tf-idf). Under this definition, "similarity" reduces to a normalized dot-product of the query vector and the document vector. So in practice, it does roughly w…

Hmm. I'll have to think about it but I don't quite see yet how this ensures that for example all documents containing all four words of a four-word query will be ranked ahead of documents containing three or fewer words of the query. Mere angle between two vectors would for example mean that a "document" consisting of three words of the query could be ranked higher than a document with thousands of words containing all four words of a query, even if the document vectors had tf-idf weighs as components. That clearly is a viable notion of similarity, but that's also clearly not what the type of search alluded to in the article is attempting to do.

Plus, your description definitely isn't something I would have been able to divine from the description from the article. Thanks for rephrasing it in comprehensible English.

Re: T-Wand: beating Lucene in less than 600 lines of code

#15

Post by codetrotter »

bryanrasmussen wrote: post #13
grumpyprole wrote: post #10

Earlier quoted context omitted.

Perhaps the OP just wanted a simple custom algorithm that would be easy to integrate? I do not understand why rejecting a million-lines-of-code dependency implies the person is a troll?

I think you are almost be definition trolling when you write beating X (millions lines of code) with 600 lines. because when you say 'beating' the implication is it does what X does better - as opposed to just matching my use case with 600 lines and not big dependency import. this usage of beating is going to rile up (i.e troll) everyone familiar with the subject who can see thousands of ways you did not beat it.

I think the heading is fine. They are beating Lucene in the context of their particular problem.

Any additional features that a product has but which you don’t need for a particular purpose is a downside, not an upside.

They’re not claiming to beat Lucene for the general case of everything Lucene is capable of.

Re: T-Wand: beating Lucene in less than 600 lines of code

#17

Post by jpgvm »

jhgb wrote: post #14
teraflop wrote: post #11

Earlier quoted context omitted.

Technically the author's description of how Lucene's ranking works is accurate. But the catch is that Lucene's default "vector space model" treats each document as a very high-dimensional vector, where the components are the words in the document (scaled using tf-idf). Under this definition, "similarity" reduces to a normalized dot-product of the query vector and the document vector. So in practice, it does roughly w…

Hmm. I'll have to think about it but I don't quite see yet how this ensures that for example all documents containing all four words of a four-word query will be ranked ahead of documents containing three or fewer words of the query. Mere angle between two vectors would for example mean that a "document" consisting of three words of the query could be ranked higher than a document with thousands of words containing a…

That is because it doesn't ensure that all documents containing all query terms are ranked above with those with a subset (though it can be tuned such that it does).

Re: T-Wand: beating Lucene in less than 600 lines of code

#18

Post by ysleepy »

The tone of the article makes it incredibly annoying and the author 100% lands on the hall of shame list of ridiculous benchmarks he mentioned.

tf-idf and phrase search alone make lucene 10x better, not speaking of the compression, fuzzy search, advanced query features etc.

I love new search engines like tantivy etc, but this is, due to the tone, a sad joke.

Re: T-Wand: beating Lucene in less than 600 lines of code

#20

Post by rory_isAdonk »

ekzhu wrote: post #3

I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that: >> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of use…

Yeah beyond the cringe of thinking a Ph.D really means anything, its just highlights the pure lack of Lucene knowledge.
Post reply on HN