Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phrase matching in Marginalia Search

marginalia.nu

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Phrase matching in Marginalia Search

#21

Post by EarlKing »

marginalia_nu wrote: post #20
outime wrote: post #19

Earlier quoted context omitted.

This comment felt like indirect spam, as it doesn't really contribute anything IMHO. The phrase "in our experience" implies they’re in the same business, and upon checking the profile, I found that aside from the bio linking to their own thing, most comments resemble (in)direct spam. Everyone has their strategies, but I really disliked seeing this.

Eh, I think it's always interesting to compare notes with other search projects. It's a small niche, and I think we're all rooting for eachother.

[flagged]

Re: Phrase matching in Marginalia Search

#22

Post by joking »

gary_0 wrote: post #3

> To make the most of phrase matching, stop words need to go. Perhaps I am misunderstanding; does this mean occurrences of stop words like "the" are stored now instead of ignored? That seems like it would add a lot of bloat. Are there any optimizations in place? Just a shot-in-the-dark suggestion, but if you are storing some bits with each keyword occurrence, can you add a few more bits to store whether the term is a…

in reality, terms are stored with the position, so if you use stopwords

to be or not to be should be indexed as 2-be 6-be and a phrase match should match exactly. the problem would be that it would match also "be not to or be", as the distance between the 2 "be" is also 3.

Long time ago it was necessary, but nowadays you loose more that what you gain using stopwords.

Re: Phrase matching in Marginalia Search

#23

Post by marginalia_nu »

arromatic wrote: post #18
marginalia_nu wrote: post #14

Earlier quoted context omitted.

1. Here are all RSS feeds known to the search engine as of some point in 2023: https://downloads.marginalia.nu/exports/feeds.csv -- it's quite noisy though, a fair number of them are anything but small web. You should be able to fetch them all in a few hours I'd reckon, and have a sample dataset to play with. There's also more data at https://downloads.marginalia.nu/exports/ , e.g. a domain level link graph, if you w…

Apologies for too many questions but resources on search engines are scarce . How do I visualize the link graphs or process them ? is there any tool preferably foss . Majestic seem to have one but it's their own .

I don't know if there's any real good answers. It's hard to visualize a graph of this size, but most graph libraries will at least consume it assuming you have a decent amount of RAM.

Re: Phrase matching in Marginalia Search

#24

Post by efilife »

I wrote my own search engine some time ago and was impressed by how well it worked on my relatively small index. And then I see this. Marginalia's dev is just unmatched with persistence and knowledge to pull all of this off, I wouldn't even know where to start some of the things he did with his search engine

Re: Phrase matching in Marginalia Search

#25

Post by marginalia_nu »

efilife wrote: post #24

I wrote my own search engine some time ago and was impressed by how well it worked on my relatively small index. And then I see this. Marginalia's dev is just unmatched with persistence and knowledge to pull all of this off, I wouldn't even know where to start some of the things he did with his search engine

To be honest it's mostly persistence. I didn't know most of this stuff when I started out, at least not as well as I do now. Having gotten the opportunity to work full time on this for a year now has also helped.

Re: Phrase matching in Marginalia Search

#26

Post by efilife »

ValleZ wrote: post #8
gary_0 wrote: post #3

> To make the most of phrase matching, stop words need to go. Perhaps I am misunderstanding; does this mean occurrences of stop words like "the" are stored now instead of ignored? That seems like it would add a lot of bloat. Are there any optimizations in place? Just a shot-in-the-dark suggestion, but if you are storing some bits with each keyword occurrence, can you add a few more bits to store whether the term is a…

Removing stop words is usually a bad advice which is beneficial only in a limited set of circumstances. Google keeps all the “the”: https://www.google.com/search?q=the

I don't think it's as simple for them as just keeping the thes. It's probably very context dependent
Post reply on HN