51–60 of 100 posts
This is very cool and very educational. Don't deploy it, though. :-) I needed something like this once, but at a little bit larger scale (few tens of thousands of documents). The answer, as always, was [sqlite]( https://www.sqlite.org/fts5.html); structurally though it's just what you have here but with someone else writing the inverted-index persistence layer.
Is it really a good idea to build something like this (big data, needs to crunch data fast) in Python (slow)?
You can see my comment about SearchArray above, but you can do a lot of native performance comparable things if you embrace array based programming
Looking at the code (src/microsearch/engine.py), we have: class SearchEngine: def __init__(self, k1: float = 1.5, b: float = 0.75): self._index: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._documents: dict[str, str] = {} self.k1 = k1 self.b = b I've no idea what `k1` or `b` are. Nor is there a single comment in the entire file. Are comments considered unfashionable these days? Looking at `_…
On mobile device but it’s the standard weighting values for either TF/IDF or BM25. In this case BM25. A comment would be useful but they are also instantly recognisable to anyone familiar with the problem.
I always love reading those when not familiar. It's almost as funny as reading something one already knows, waiting for the punch line...
What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.
Although it's not formal, my team sometimes says "this code is not grug" or "this code is pretty grug" in reference to https://grugbrain.dev
I believe there's a saying about a woodchuck somewhere in here
Looking at the code (src/microsearch/engine.py), we have: class SearchEngine: def __init__(self, k1: float = 1.5, b: float = 0.75): self._index: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._documents: dict[str, str] = {} self.k1 = k1 self.b = b I've no idea what `k1` or `b` are. Nor is there a single comment in the entire file. Are comments considered unfashionable these days? Looking at `_…
Names can convey proper semantic information about what your program does, use them godammit
At least only some of the time, unfortunately. What power users want is "grep for the Web", not "Google, tell me what you want me to see."
Earlier quoted context omitted.
Thanks for your comment Doug! I just bought your book (Relevance Search) and I'm planning to read it asap. I'll give a look to your project and try it for experimenting, thanks for sharing.
I also bought the book (it’s Relevant not Relevance BTW), that one and Deep Learning for Search were invaluable when building a news search engine a few years ago (right before and in the middle of the pandemic).
It's one I point people at all the time when they ask me why something isn't working as expected in any standard search tool, and something I reference from time to time to refresh my own knowledge.
Well worth the money.
Earlier quoted context omitted.
HN has an RSS feed [0] so there's no need to keep refreshing or make the rounds of this and other sites that have interesting information. I have my own feed setup with sites I like to frequent [1]. [0] https://hackaday.com/blog/feed/ [1] https://mechaelephant.com/feed
what software did you use for your own feed?
[0] https://github.com/abetusk/www.mechaelephant.com/tree/releas...