A search engine in 80 lines of Python
alexmolas.com
A search engine in 80 lines of Python
1–10 of 100 posts
Re: A search engine in 80 lines of Python
#2This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing.
https://github.com/softwaredoug/searcharray
Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy...
BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you want to compact positions into as little memory as possible
https://github.com/softwaredoug/searcharray/blob/main/search...
Re: A search engine in 80 lines of Python
#3This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy... BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you…
Thanks for your comment Doug! I just bought your book (Relevance Search) and I'm planning to read it asap.
I'll give a look to your project and try it for experimenting, thanks for sharing.
Re: A search engine in 80 lines of Python
#4This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy... BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you…
Have you found that including sentiment analysis has helped(or hurt) with phrases? I also have found that phrases are difficult to work with and I’m wondering what I can do to improve performance.
Re: A search engine in 80 lines of Python
#5This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy... BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you…
How come you found this post and commented on it so quickly, do you have some sort of search scanning frontpage for terms of interest or was it by chance? Curious
Re: A search engine in 80 lines of Python
#6This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy... BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you…
How come you found this post and commented on it so quickly, do you have some sort of search scanning frontpage for terms of interest or was it by chance? Curious
Huh? I check Hacker News multiple times a day - it's not odd to click on an article within an hour of it being posted.
Re: A search engine in 80 lines of Python
#7Earlier quoted context omitted.
How come you found this post and commented on it so quickly, do you have some sort of search scanning frontpage for terms of interest or was it by chance? Curious
Huh? I check Hacker News multiple times a day - it's not odd to click on an article within an hour of it being posted.
It's not the first time I saw an article posted and then an expert in the field comment on it rather quickly, I thought I may be missing something how other people use this site, had no negative intentions asking this and thanks for the answer ;)
Re: A search engine in 80 lines of Python
#8Checks out. Most of the difficulty in search is dealing with the data volume. The logic itself is surprisingly easy, or it can be. Of course you can complicate it endlesssly, but this project successfully cuts most of the fluff...
So with that in mind, approaching the problem not as one of how to make the search engine bigger, but the data smaller (physically or better signal/noise ratio) goes a very long way.
Re: A search engine in 80 lines of Python
#9Earlier quoted context omitted.
Huh? I check Hacker News multiple times a day - it's not odd to click on an article within an hour of it being posted.
It's not the first time I saw an article posted and then an expert in the field comment on it rather quickly, I thought I may be missing something how other people use this site, had no negative intentions asking this and thanks for the answer ;)
HN has an RSS feed [0] so there's no need to keep refreshing or make the rounds of this and other sites that have interesting information.
I have my own feed setup with sites I like to frequent [1].
Re: A search engine in 80 lines of Python
#10Thanks for this lovely well written account. Made me wish I still
taught this kind of coding because it would be such a lovely example
for students to work through. (We used to do a bootcamp of build your
own Facebook and Google in a week) Very inspirational.