Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

T-Wand: beating Lucene in less than 600 lines of code

yyhh.org

Post reply on HN
Print view View original

31–40 of 96 posts

Re: T-Wand: beating Lucene in less than 600 lines of code

#31

Post by softwaredoug »

ekzhu wrote: post #3

I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that: >> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of use…

As author of Relevant Search and contriburor of AI powered search I endorse this :)

Relevance is really subjective, domain specific, requires intense amount of measurement and testing and many different ranking signals. Lucene is a toolbox for crafting many of these signals.

Re: T-Wand: beating Lucene in less than 600 lines of code

#32

Post by Cavallium »

ptspts wrote: post #29

As a user, when I write a search query as a space-separated list of n words, I expect to see documents which contain all of those words (or synonyms). I absolutely don't want to see documents which contain only n-1 of the specified words. I know exactly what I want, and the reason I've added the last word to the query is that there were too many generic, boring, irrelevant results without it.

You completely misunderstood what this project does. This kind of libraries do not force you to return results in a specific way. Generally they allow the programmer to customize how the results are scored and chosen.

Re: T-Wand: beating Lucene in less than 600 lines of code

#33

Post by marcellus23 »

speedgoose wrote: post #26
rory_isAdonk wrote: post #20

Earlier quoted context omitted.

Yeah beyond the cringe of thinking a Ph.D really means anything, its just highlights the pure lack of Lucene knowledge.

A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.

Well it’s certainly more meaningful to the person who has one.

Re: T-Wand: beating Lucene in less than 600 lines of code

#35

Post by Cthulhu_ »

speedgoose wrote: post #26
rory_isAdonk wrote: post #20

Earlier quoted context omitted.

Yeah beyond the cringe of thinking a Ph.D really means anything, its just highlights the pure lack of Lucene knowledge.

A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.

Well yeah, but it shouldn't make you smug. A Ph.D, while not the beginning, is definitely not the end, and the people we look up to as Really Smart never sat on their laurels and quit learning, never considered themselves above others.

Re: T-Wand: beating Lucene in less than 600 lines of code

#36

Post by ibejoeb »

Tangential, but it reminds me of GNU parallel's man page example one-liner for regex-matching giant texts, which goes on to say:

> If the problem is too big to be solved by this, you are probably ready for Lucene.

https://www.gnu.org/software/parallel/parallel.html?highligh...

Re: T-Wand: beating Lucene in less than 600 lines of code

#37

Post by kordlessagain »

Every single document “page” humans index has three “interesting” dimensions: the height and width of the page the words appear on, plus another for the font size.

So, what is a document anyway? Is a file handle just another type of dimension? Or, is a picture on the page part of the page document, or is it its own document?

Humans use arbitrary dimensions to tokenize the document’s bits, so to speak.

My dad could read a book (a big document) in blocks or chunks. In thinking of it that way it become apparent that groups and positions of data may constitute their own document boundaries and that info may be used in both the indexing and search pipelines to build relevance and relatedness.

Re: T-Wand: beating Lucene in less than 600 lines of code

#38

Post by The_rationalist »

teraflop wrote: post #11
jhgb wrote: post #8

> Most search engines use something called a vector space model, where both user queries and documents are reduced to vectors (i.e. a fixed number of numbers). That is to say, the search engines are not looking at the meanings of the queries or the documents. Instead, they turned them both into some numbers. The search problem, is reduced to a problem of finding the similarity between the numbers representing the que…

Technically the author's description of how Lucene's ranking works is accurate. But the catch is that Lucene's default "vector space model" treats each document as a very high-dimensional vector, where the components are the words in the document (scaled using tf-idf). Under this definition, "similarity" reduces to a normalized dot-product of the query vector and the document vector. So in practice, it does roughly w…

isn't BM25 used by default instead of tf-idf?

Re: T-Wand: beating Lucene in less than 600 lines of code

#39

Post by juancb »

Cthulhu_ wrote: post #35
speedgoose wrote: post #26

Earlier quoted context omitted.

A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.

Well yeah, but it shouldn't make you smug. A Ph.D, while not the beginning, is definitely not the end, and the people we look up to as Really Smart never sat on their laurels and quit learning, never considered themselves above others.

It's fascinating seeing what hacker news thinks is smug. There are plenty of actually smug comments that go without being called out but for some reason this guy does.

The PhD flex in jest by the blog post's author is a bit awkward but I don't know that I would characterize it as smug.

Any insight into why the author is perceived as smug would be appreciated.

Re: T-Wand: beating Lucene in less than 600 lines of code

#40

Post by johncearls »

tgv wrote: post #30
speedgoose wrote: post #26

Earlier quoted context omitted.

A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.

But a phd in HCI has very little relevance to information retrieval or computational linguistics.

Exactly this. I have a PHD in CS and am a world class expert in multi-omic data integration and analysis. I'm happy to throw my weight around in that area, but I'd never point to my PhD to pontificate on Neural nets or systems or queuing theory or 99 percent of CS. If getting a PhD doesn't teach you how much you don't know and how hard it is to develop real expertise in any area, I think you wasted your PhD. Note, my PhD isn't on the value of PhDs so take it as you will.
Post reply on HN