41–50 of 100 posts
What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.
Although it's not formal, my team sometimes says "this code is not grug" or "this code is pretty grug" in reference to https://grugbrain.dev
This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy... BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you…
Thanks for your comment Doug! I just bought your book (Relevance Search) and I'm planning to read it asap. I'll give a look to your project and try it for experimenting, thanks for sharing.
What is the point of flexing about LOC, if it is not a total number of \r\n since we are using external deps? I know that there is no unit for codebase in SI system, but I think we should measure cognitive load somehow.
But even then, those libraries aren't related to search engines. If we start counting generic dependencies like pandas and fastapi, we might as well start counting the millions of loc of the operating system needed to run this search engine, and the firmware in the network card. Maybe even account for the complexity of the hardware this is running on.
Earlier quoted context omitted.
Huh? I check Hacker News multiple times a day - it's not odd to click on an article within an hour of it being posted.
It's not the first time I saw an article posted and then an expert in the field comment on it rather quickly, I thought I may be missing something how other people use this site, had no negative intentions asking this and thanks for the answer ;)
Also, a lot of subject matter experts hang out here so it just happens a lot :)
class SearchEngine:
def __init__(self, k1: float = 1.5, b: float = 0.75):
self._index: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int))
self._documents: dict[str, str] = {}
self.k1 = k1
self.b = b
Looking at `_documents`, I'm guessing the keys are URLs and the values contents of those URLs, but i might be wrong.
The whole thing looks like it would've been a useful resource for people to learn how to build search engines with, that people could build on, had the writer been bothered to document it. But as it is I'm disappointed with the poor code.
Looking at the code (src/microsearch/engine.py), we have: class SearchEngine: def __init__(self, k1: float = 1.5, b: float = 0.75): self._index: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._documents: dict[str, str] = {} self.k1 = k1 self.b = b I've no idea what `k1` or `b` are. Nor is there a single comment in the entire file. Are comments considered unfashionable these days? Looking at `_…
A comment would be useful but they are also instantly recognisable to anyone familiar with the problem.
Looking at the code (src/microsearch/engine.py), we have: class SearchEngine: def __init__(self, k1: float = 1.5, b: float = 0.75): self._index: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._documents: dict[str, str] = {} self.k1 = k1 self.b = b I've no idea what `k1` or `b` are. Nor is there a single comment in the entire file. Are comments considered unfashionable these days? Looking at `_…
Is it really a good idea to build something like this (big data, needs to crunch data fast) in Python (slow)?
And once you do, odds are you'd find most of the runtime spent in very small portions of code doing fairly basic composable operations on large compressed arrays, and you can get very far with just rewriting a tiny core in something faster.
The number of people who need a scale where this is hard to do fast enough is small...