Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine in 80 lines of Python

alexmolas.com

Post reply on HN
Print view View original

21–30 of 100 posts

Re: A search engine in 80 lines of Python

#21

Post by marginalia_nu »

renegat0x0 wrote: post #18

I have myself dabbled a little bit in that subject. Some of my notes: - some RSS feeds are protected by cloudflare. It is true however that it is not necessary for majority of blogs. If you would like to do more then selenium would be a way to solve "cloudflare" protected links - sometimes even selenium headless is not enough and full blown browser in selenium is necessary to fool it's protection - sometimes even tha…

> I wanted also to process domains from https://downloads.marginalia.nu/, but haven't got time to read structure of the files

Feel free to shoot me an email if you have any questions about how to use the files. They're very much there to fan the flames of other indie search projects :D

Re: A search engine in 80 lines of Python

#22

Post by pphysch »

smalu wrote: post #15

What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.

It's meaningful here because if it said "A search engine in 4000 lines of Python" most readers' eyes would glaze over, but 80 is short enough to warrant a glance.

Re: A search engine in 80 lines of Python

#23

Post by smalu »

hobs wrote: post #19
smalu wrote: post #15

What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.

Well, the old school way is https://en.wikipedia.org/wiki/Cyclomatic_complexity

I am aware of it, but as far as I understand, cyclomatic complexity is used to compare implementations of single flows (algorithms), not for codebases. What is the cyclomatic complexity of Facebook? :)

Re: A search engine in 80 lines of Python

#24

Post by dharmab »

smalu wrote: post #15

What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.

Although it's not formal, my team sometimes says "this code is not grug" or "this code is pretty grug" in reference to https://grugbrain.dev

Re: A search engine in 80 lines of Python

#25

Post by eichin »

behnamoh wrote: post #13

Is it really a good idea to build something like this (big data, needs to crunch data fast) in Python (slow)?

Yes. Paragraph 3:

> This implementation doesn’t pretend to be a production-ready search engine, just a usable toy example showing how a search engine works under the hood.

The whole point is expressiveness.

Re: A search engine in 80 lines of Python

#26

Post by DontchaKnowit »

dharmab wrote: post #24
smalu wrote: post #15

What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.

Although it's not formal, my team sometimes says "this code is not grug" or "this code is pretty grug" in reference to https://grugbrain.dev

Thats awesome im stealing this

Re: A search engine in 80 lines of Python

#27

Post by lijok »

smalu wrote: post #15

What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.

Why not celebrate the achievements of the industry allowing us to build something like this in 80 LOC?

Re: A search engine in 80 lines of Python

#28

Post by softwaredoug »

karolist wrote: post #5
softwaredoug wrote: post #2

This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy... BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you…

How come you found this post and commented on it so quickly, do you have some sort of search scanning frontpage for terms of interest or was it by chance? Curious

I got that HN gaaaame

Re: A search engine in 80 lines of Python

#29

Post by golergka »

smalu wrote: post #15

What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.

Operating system and programming language are also external dependencies.

Re: A search engine in 80 lines of Python

#30

Post by Drakim »

lijok wrote: post #27
smalu wrote: post #15

What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.

Why not celebrate the achievements of the industry allowing us to build something like this in 80 LOC?

I mean, I could import this achievement in my own project and build a search engine in 1 LOC.
Post reply on HN