Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phrase matching in Marginalia Search

marginalia.nu

Post reply on HN
Print view View original

1–10 of 26 posts

Re: Phrase matching in Marginalia Search

#2

Post by ColinHayhurst »

Congrats Viktor.

> The feedback cycle in web search engine development is very long....Overall the approach taken to improving search result quality is looking at a query that does not give good results, asking what needs to change for that to improve, and then making that change. Sometimes it’s a small improvement, sometimes it’s a huge game changer.

Yes, this resonates with our experience

Re: Phrase matching in Marginalia Search

#3

Post by gary_0 »

> To make the most of phrase matching, stop words need to go.

Perhaps I am misunderstanding; does this mean occurrences of stop words like "the" are stored now instead of ignored? That seems like it would add a lot of bloat. Are there any optimizations in place?

Just a shot-in-the-dark suggestion, but if you are storing some bits with each keyword occurrence, can you add a few more bits to store whether the term is adjacent to a common stop word? So maybe if you have to=0 or=1, "to be or not to be" would be able to match the data `0be 1not 0be`, where only "be" and "not" are actual keywords. But the extra metadata bits can be ignored, so pages containing "The Clash" will match both the literal query (via the "the" bit), and just "clash" (without the "the" bit).

Re: Phrase matching in Marginalia Search

#5

Post by marginalia_nu »

gary_0 wrote: post #3

> To make the most of phrase matching, stop words need to go. Perhaps I am misunderstanding; does this mean occurrences of stop words like "the" are stored now instead of ignored? That seems like it would add a lot of bloat. Are there any optimizations in place? Just a shot-in-the-dark suggestion, but if you are storing some bits with each keyword occurrence, can you add a few more bits to store whether the term is a…

It's not as bad as you might think, we're speaking dozens of GB across the entire index.

I don't think stopwords as an optimization makes sense when you go beyond BM25. The search engine behaves worse and adding a bunch of optimizations makes an already incrediby complex piece of software more so.

So overall I don't think the juice is worth the squeeze.

Re: Phrase matching in Marginalia Search

#8

Post by ValleZ »

gary_0 wrote: post #3

> To make the most of phrase matching, stop words need to go. Perhaps I am misunderstanding; does this mean occurrences of stop words like "the" are stored now instead of ignored? That seems like it would add a lot of bloat. Are there any optimizations in place? Just a shot-in-the-dark suggestion, but if you are storing some bits with each keyword occurrence, can you add a few more bits to store whether the term is a…

Removing stop words is usually a bad advice which is beneficial only in a limited set of circumstances. Google keeps all the “the”: https://www.google.com/search?q=the

Re: Phrase matching in Marginalia Search

#9

Post by heikkilevanto »

gary_0 wrote: post #3

> To make the most of phrase matching, stop words need to go. Perhaps I am misunderstanding; does this mean occurrences of stop words like "the" are stored now instead of ignored? That seems like it would add a lot of bloat. Are there any optimizations in place? Just a shot-in-the-dark suggestion, but if you are storing some bits with each keyword occurrence, can you add a few more bits to store whether the term is a…

One of the problems with stop words is that they vary between languages. "The" is a good candidate in English, but in Danish it just means "tea", which should be a valid search term. And even in English, what looks like a serious stop word, can be an integral part of the phrase. "How to use The in English".
Post reply on HN