T-Wand: beating Lucene in less than 600 lines of code
1–10 of 96 posts
Re: T-Wand: beating Lucene in less than 600 lines of code
#2This is part of Datalevin, an interesting open source Datalog database: https://github.com/juji-io/datalevin
Re: T-Wand: beating Lucene in less than 600 lines of code
#3I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF.
> As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that:
>> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of user query terms.
> This makes perfect sense. Right?
Also, “most search engines” don’t use vector space model as the only way to rank result, for example, page rank.
Edit: in some search scenarios finding the documents with the most query terms make sense, but Lucene can also rank using this metric. Still, applaud the author's effort in digging into research literature. Search relevance is very hard and standard off the shelf metrics like TF-IDF and page rank are often not enough. Good search usually requires deep understanding of the specific subject domain and hand-tuning tons of signals, many of which aren't even strictly based on search terms (e.g., previously purchased products on a store's website, geographic location, trending results).
Re: T-Wand: beating Lucene in less than 600 lines of code
#4I'm not sure if this is a troll. It seems the author did not spend much time learning Lucene, it already does everything he needs and better. Of course Lucene does tf-idf! If you are not getting the results you want, you must not be indexing right or must have something misconfigured. Lucene lets you tweak relevance, speed, index space efficiency, etc in a million ways.
Re: T-Wand: beating Lucene in less than 600 lines of code
#5Interesting. I can't say Lucene gives great results for relevance on a large database, eg. Confluence.
However there is a large space between algorithmic assumptions and what "relevance" actually is to the user. This article may improve, but my feeling is -- both for search and for the article -- that more context is needed.
Re: T-Wand: beating Lucene in less than 600 lines of code
#6Lucene has the concept of "minimum match," [1] which is what you're playing with. You can implement your algorithm on top of this relatively trivially, by re-querying with min_match=n, min_match=n-1, etc until you get results. If you wrote a Lucene plugin to do this, it would also be small (People have chosen not to do this because (1) it adds complexity, and (2) the bold declaration you made (A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of user query terms) turns out to be only somewhat true in practice. It's pretty common for pagerank and other non-linguistic ranking signals etc to trump linguistic preciseness.
That said, everything is a tradeoff, and the industry has clearly chosen to return more candidates than strictly necessary, then prune them after the fact with better ML & relevance, rather than trying to construct the perfectly minimal candidate set while missing some possible results.
[1] See https://www.elastic.co/guide/en/elasticsearch/reference/curr...
Re: T-Wand: beating Lucene in less than 600 lines of code
#7Interesting. I can't say Lucene gives great results for relevance on a large database, eg. Confluence. However there is a large space between algorithmic assumptions and what "relevance" actually is to the user. This article may improve, but my feeling is -- both for search and for the article -- that more context is needed.
Lucene is extremely flexible. You have to configure it to do what you want if you don't like the defaults (which OP didn't do).
Re: T-Wand: beating Lucene in less than 600 lines of code
#8> Most search engines use something called a vector space model, where both user queries and documents are reduced to vectors (i.e. a fixed number of numbers). That is to say, the search engines are not looking at the meanings of the queries or the documents. Instead, they turned them both into some numbers. The search problem, is reduced to a problem of finding the similarity between the numbers representing the query and the numbers representing the documents.
I don't see how this is true for concordance-like searches (Trados-like, which seems to be what the article is concerned with - preferring the documents with all query words present in it, otherwise the documents with the most query words present in it, etc.). This sounds more like description of similarity useful for metric space indices or something like that.
Re: T-Wand: beating Lucene in less than 600 lines of code
#9The author deeply misunderstands how WAND works, and "T-WAND" is just WAND. Once a pivot is found, you don't have to advance all the iterators that precede it, but just enough to disprove the current document, at which point a new pivot is chosen (this is equivalent to the heuristic described in the article).
The order in which iterators are tested against the pivot can be term cardinality ordering (which is one of the heuristics described in the original paper AFAIR). Then, the pivot documents are a subset of what the article calls the current "tier" (union of the k rarest terms).
Re: T-Wand: beating Lucene in less than 600 lines of code
#10I'm not sure if this is a troll. It seems the author did not spend much time learning Lucene, it already does everything he needs and better. Of course Lucene does tf-idf! If you are not getting the results you want, you must not be indexing right or must have something misconfigured. Lucene lets you tweak relevance, speed, index space efficiency, etc in a million ways.
Perhaps the OP just wanted a simple custom algorithm that would be easy to integrate?
I do not understand why rejecting a million-lines-of-code dependency implies the person is a troll?