Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine in 80 lines of Python

alexmolas.com

Post reply on HN
Print view View original

31–40 of 100 posts

Re: A search engine in 80 lines of Python

#31

Post by Xenoamorphous »

alexmolas wrote: post #3
softwaredoug wrote: post #2

This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy... BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you…

Thanks for your comment Doug! I just bought your book (Relevance Search) and I'm planning to read it asap. I'll give a look to your project and try it for experimenting, thanks for sharing.

I also bought the book (it’s Relevant not Relevance BTW), that one and Deep Learning for Search were invaluable when building a news search engine a few years ago (right before and in the middle of the pandemic).

Re: A search engine in 80 lines of Python

#34

Post by MultiVectors »

Really cool project! Building a search engine from the ground up to tackle small site discoverability? That's a challenge I can get behind. I'm especially into how you used asyncio in Python for faster crawling – smart move.

But, let's talk scale and features. As it stands, handling bigger data sets or adding more complex search features might be tough. On the features side, playing around with query operators or n-gram indexing could seriously level up your search results.

Expanding beyond RSS for content could also give your engine a nice touch. Just throwing in my thoughts – been around the block with search tech a bit. Can't wait to see what's next for your project!

Re: A search engine in 80 lines of Python

#36

Post by duck »

Drakim wrote: post #30
lijok wrote: post #27

Earlier quoted context omitted.

Why not celebrate the achievements of the industry allowing us to build something like this in 80 LOC?

I mean, I could import this achievement in my own project and build a search engine in 1 LOC.

You code just use an existing search engine and it would be 0 LOC, but I think you're missing the point. The focus wasn't on 80 LOC, but rather being able to talk through it in a short blog post.

Re: A search engine in 80 lines of Python

#37

Post by FergusArgyll »

dharmab wrote: post #24
smalu wrote: post #15

What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.

Although it's not formal, my team sometimes says "this code is not grug" or "this code is pretty grug" in reference to https://grugbrain.dev

Best thing I've read today, thanks!

Re: A search engine in 80 lines of Python

#38

Post by tsukurimashou »

abetusk wrote: post #9
karolist wrote: post #7

Earlier quoted context omitted.

It's not the first time I saw an article posted and then an expert in the field comment on it rather quickly, I thought I may be missing something how other people use this site, had no negative intentions asking this and thanks for the answer ;)

HN has an RSS feed [0] so there's no need to keep refreshing or make the rounds of this and other sites that have interesting information. I have my own feed setup with sites I like to frequent [1]. [0] https://hackaday.com/blog/feed/ [1] https://mechaelephant.com/feed

what software did you use for your own feed?

Re: A search engine in 80 lines of Python

#39

Post by hombre_fatal »

Drakim wrote: post #30
lijok wrote: post #27

Earlier quoted context omitted.

Why not celebrate the achievements of the industry allowing us to build something like this in 80 LOC?

I mean, I could import this achievement in my own project and build a search engine in 1 LOC.

"X in N lines" is interesting because it's going to show you the minimal implementation of the interesting or central part of the solution without all the other moving parts of a production system, usually for the purpose of demonstrating how something works.

You can see how your 1-liner pitch doesn't fulfill this expectation.

Re: A search engine in 80 lines of Python

#40

Post by adw »

This is very cool and very educational. Don't deploy it, though. :-)

I needed something like this once, but at a little bit larger scale (few tens of thousands of documents). The answer, as always, was [sqlite](https://www.sqlite.org/fts5.html); structurally though it's just what you have here but with someone else writing the inverted-index persistence layer.

Post reply on HN