Imo, it's not fair to talk about 80 lines of code while using thitd party libraries (feedparser, bs4, etc)
[dead]
A search engine in 80 lines of Python
71–80 of 100 posts
Re: A search engine in 80 lines of Python
#72Earlier quoted context omitted.
this trend of `a: float` always reminds me of the Rich Hickey "you don't want types, you want proper names" talk. I really hate this (feels to me Go inspired) tendency of undescriptive single letter variable, with the type system abused as a naming assistant. Names can convey proper semantic information about what your program does, use them godammit
> tendency of undescriptive single letter variable There are 2 schools of thought on which one is clearer, F = G * m1 * m2 / r**2 or force = gravitational_constant * mass_of_body_1 * mass_of_body_2 / distance_between_bodies ** 2
Phycisist :
> F = G * m1 * m2 / r**2
Computer scientist:
nonrelativistic_gravitational_force = (
Physics.NonRelativistic.Gravity.gravitational_constant
* body1.NonRelativistic.mass()
* body2.NonRelativistic.mass()
/ body1.NonRelativistic.distanceTo(body2) ** 2
)*Re: A search engine in 80 lines of Python
#73Google allows you to search for "search engine" (notice the double quotes) and it’ll only show you results where the two words appear in this specific order. At least only some of the time, unfortunately. What power users want is "grep for the Web", not "Google, tell me what you want me to see."
> What power users want is "grep for the Web", not "Google, tell me what you want me to see."
I can almost guarantee that nobody actually wants this. "Grep for the web" is strictly bad compared to a search engine that does the tiniest amount of query expansion. Google is definitely taking too many liberties in interpreting the query, but there are many things any search engine should do that will be a straight improvement over not doing them.
The problem with Google search right now is that it's hard to reason about why it gives the results it does, seemingly because they rely too heavily on embeddings to compare strings. It's frustrating when "cat food" matches "dog restaurant" because the two were semantically close in some embedding space that doesn't quite align with human reasoning.
Re: A search engine in 80 lines of Python
#74Don't use keywords (1-grams), the best results for English can be achieved with 2+3-grams. n-grams retain context.
I think you'd probably get the best result with both. There's definitely real merit to a keyword-understanding of which terms appear in the title or as a named entity for example.
Re: A search engine in 80 lines of Python
#75Imo, it's not fair to talk about 80 lines of code while using thitd party libraries (feedparser, bs4, etc)
If they'd built it on top of elasticsearch I'd agree with this sentiment, but given the actual search engine bit is implemented in those 80 lines, I think it's fair. The libraries being pulled are exactly the sort of libraries you shouldn't try to hand-roll.
(sometimes you see those articles like 'built your own search engine' and it's a guide on how to install searxng or yacy or something.)
Re: A search engine in 80 lines of Python
#76Earlier quoted context omitted.
> tendency of undescriptive single letter variable There are 2 schools of thought on which one is clearer, F = G * m1 * m2 / r**2 or force = gravitational_constant * mass_of_body_1 * mass_of_body_2 / distance_between_bodies ** 2
Phycisist : > F = G * m1 * m2 / r**2 Computer scientist: nonrelativistic_gravitational_force = ( Physics.NonRelativistic.Gravity.gravitational_constant * body1.NonRelativistic.mass() * body2.NonRelativistic.mass() / body1.NonRelativistic.distanceTo(body2) ** 2 )*
Software engineer straight out of uni:
# Get force of gravity as below to be used later
F = G * m1 * m2 / r*2
Software engineer 5 years after uni:
gravitational_force = (
PhysicsContextConstructorFactory.createByRelativisticEnum(SystemConfig.getRelativisticEnum()).construct().getGravitationalConstant().value()
* body1.getRelativisticContext(SystemConfig.getRelativisticEnum()).mass().value()
* body2.getRelativisticContext(SystemConfig.getRelativisticEnum()).mass().value()
/ SystemConfig.getRelativisticContext(SystemConfig.getRelativisticEnum()).distanceTo(body1, body2).value() \* PlatformUnsignedInt(PlatformMinMaxAwareUnsignedInt(2).value()).getUnsignedInt().value()
)*
# Newton's law of universal gravitation is well within wanted margin of error
F = G * m1 * m2 / r*2
Re: A search engine in 80 lines of Python
#77Earlier quoted context omitted.
I mean, I could import this achievement in my own project and build a search engine in 1 LOC.
You code just use an existing search engine and it would be 0 LOC, but I think you're missing the point. The focus wasn't on 80 LOC, but rather being able to talk through it in a short blog post.
I don't think that LOC affects one's ability to effectively communicate the way their code operates. And, if we really need to, lowering the amount of operations required to attain the desired result would be better than lowering lined of code, sine low ops = less explanation.
Re: A search engine in 80 lines of Python
#78What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.
Although it's not formal, my team sometimes says "this code is not grug" or "this code is pretty grug" in reference to https://grugbrain.dev
me nod head save to pdf not just bookmark link
Re: A search engine in 80 lines of Python
#79Earlier quoted context omitted.
It's meaningful here because if it said "A search engine in 4000 lines of Python" most readers' eyes would glaze over, but 80 is short enough to warrant a glance.
"A search engine in 80 columns of Python code!"
"A search engine in 80 lines of Python code" (but every line is 10 statements separated by semicolon)
Re: A search engine in 80 lines of Python
#80Looking at the code (src/microsearch/engine.py), we have: class SearchEngine: def __init__(self, k1: float = 1.5, b: float = 0.75): self._index: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._documents: dict[str, str] = {} self.k1 = k1 self.b = b I've no idea what `k1` or `b` are. Nor is there a single comment in the entire file. Are comments considered unfashionable these days? Looking at `_…
Hi, author here. If I wanted a catchy title for the post I needed to cut the number of LOC as much as possible;) Joking apart, thanks for your feedback. I agree that usually it's better to have documentation and code together, but in this case since it's an educational project I decided to split code and documentation, and document the code in a blog post.
Fair enough