A search engine in 80 lines of Python • Hacker News Board

Re: A search engine in 80 lines of Python

#81

Post by cabalamat » Thu, Feb 08, 2024, 12:05 PM UTC

sampo wrote: post #65

Barrin92 wrote: post #56

Earlier quoted context omitted.

this trend of `a: float` always reminds me of the Rich Hickey "you don't want types, you want proper names" talk. I really hate this (feels to me Go inspired) tendency of undescriptive single letter variable, with the type system abused as a naming assistant. Names can convey proper semantic information about what your program does, use them godammit

> tendency of undescriptive single letter variable There are 2 schools of thought on which one is clearer, F = G * m1 * m2 / r**2 or force = gravitational_constant * mass_of_body_1 * mass_of_body_2 / distance_between_bodies ** 2

Short names are OK if they are explained somewhere. I often explain them in a comment at the start of a class or function.

Re: A search engine in 80 lines of Python

#82

Post by lloeki » Thu, Feb 08, 2024, 12:13 PM UTC

ZaoLahma wrote: post #76

trashtester wrote: post #72

Earlier quoted context omitted.

Phycisist : > F = G * m1 * m2 / r**2 Computer scientist: nonrelativistic_gravitational_force = ( Physics.NonRelativistic.Gravity.gravitational_constant * body1.NonRelativistic.mass() * body2.NonRelativistic.mass() / body1.NonRelativistic.distanceTo(body2) ** 2 )*

Software engineer straight out of uni: # Get force of gravity as below to be used later F = G * m1 * m2 / r*2 Software engineer 5 years after uni: gravitational_force = ( PhysicsContextConstructorFactory.createByRelativisticEnum(SystemConfig.getRelativisticEnum()).construct().getGravitationalConstant().value() * body1.getRelativisticContext(SystemConfig.getRelativisticEnum()).mass().value() * body2.getRelativisticCon…

Sadly the 5y SE forgot about dependency injecting the computation... after all we're always on the brink of new physics discovery, we'd hate to have to rewrite all that code when we can readily swap in fresh laws with the proper architecture!

Re: A search engine in 80 lines of Python

#83

Post by softwaredoug » Thu, Feb 08, 2024, 12:36 PM UTC

alpinelogic wrote: post #70

softwaredoug wrote: post #2

This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy... BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you…

Hey, I tackled phrase matching in my toy project here: https://github.com/vasilionjea/lofi-dx/blob/main/test/search... I think I tested it thoroughly but any feedback would be appreciated! Edit: I delta-encoded and base36-encoded the positions

Oh nice I’ll take a look!

Re: A search engine in 80 lines of Python

#84

Post by pshirshov » Thu, Feb 08, 2024, 12:48 PM UTC

marginalia_nu wrote: post #74

pshirshov wrote: post #46

Don't use keywords (1-grams), the best results for English can be achieved with 2+3-grams. n-grams retain context.

I think you'd probably get the best result with both. There's definitely real merit to a keyword-understanding of which terms appear in the title or as a named entity for example.

Usually you would want to have weighted n-grams with 1-grams having lowest weights. In many cases it's better to have zeros. For English, 4-grams react on generic phrases/idioms, 5+ are way too selective and just keywords usually reduce relevance too much. 2+3 are the best.

Re: A search engine in 80 lines of Python

#85

Post by pshirshov » Thu, Feb 08, 2024, 12:50 PM UTC

Also, forgot to add, for this approach you better use sketches (bloom filter/hyperminhash). Same results with lightning performance.

Re: A search engine in 80 lines of Python

#86

Post by ZaoLahma » Thu, Feb 08, 2024, 1:14 PM UTC

lloeki wrote: post #82

ZaoLahma wrote: post #76

Earlier quoted context omitted.

Software engineer straight out of uni: # Get force of gravity as below to be used later F = G * m1 * m2 / r*2 Software engineer 5 years after uni: gravitational_force = ( PhysicsContextConstructorFactory.createByRelativisticEnum(SystemConfig.getRelativisticEnum()).construct().getGravitationalConstant().value() * body1.getRelativisticContext(SystemConfig.getRelativisticEnum()).mass().value() * body2.getRelativisticCon…

Sadly the 5y SE forgot about dependency injecting the computation... after all we're always on the brink of new physics discovery, we'd hate to have to rewrite all that code when we can readily swap in fresh laws with the proper architecture!

Oh, but that comes as a needs-to-be-fixed to the pull request from the 6y SE, so no worries at all.

Re: A search engine in 80 lines of Python

#87

Post by burning_hamster » Thu, Feb 08, 2024, 1:20 PM UTC

dharmab wrote: post #24

smalu wrote: post #15

What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.

Although it's not formal, my team sometimes says "this code is not grug" or "this code is pretty grug" in reference to https://grugbrain.dev

This grug not big brain. This grug ate food and read article on big screen at the same time. Now food on screen, not in belly. Grug still hungry, but need to find rag now to clean screen. Otherwise no more work and no more shiny rocks for grug. But grug thanks other grug for article. Grug belly empty, but grug brain full now.

Re: A search engine in 80 lines of Python

#88

Post by make3 » Thu, Feb 08, 2024, 1:32 PM UTC

just fyi, PageRank and BM25 address orthogonal problems of reputation/popularity (PageRank) and query/semantic matching (BM25). Saying you're using bm25 instead of PageRank makes no sense.

Re: A search engine in 80 lines of Python

#89

Post by make3 » Thu, Feb 08, 2024, 1:35 PM UTC

malomalsky wrote: post #67

Imo, it's not fair to talk about 80 lines of code while using thitd party libraries (feedparser, bs4, etc)

it is if they're extremely general & main stream

Re: A search engine in 80 lines of Python

#90

Post by marginalia_nu » Thu, Feb 08, 2024, 1:57 PM UTC

cabalamat wrote: post #47

Looking at the code (src/microsearch/engine.py), we have: class SearchEngine: def __init__(self, k1: float = 1.5, b: float = 0.75): self._index: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._documents: dict[str, str] = {} self.k1 = k1 self.b = b I've no idea what `k1` or `b` are. Nor is there a single comment in the entire file. Are comments considered unfashionable these days? Looking at `_…

k1 and b are tuning parameters for the BM-25 ranking function[1]. These are not names OP's invented. Virtually every implementation and definitely every textbook uses these variable names. You give them the name k1 and b because otherwise nobody who's into information retrieval will understand what they do.

[1] https://en.wikipedia.org/wiki/Okapi_BM25