Why not use whoosh? https://whoosh.readthedocs.io/en/latest/quickstart.html
A search engine in 80 lines of Python
91–100 of 100 posts
Re: A search engine in 80 lines of Python
#92Earlier quoted context omitted.
Well, the old school way is https://en.wikipedia.org/wiki/Cyclomatic_complexity
I am aware of it, but as far as I understand, cyclomatic complexity is used to compare implementations of single flows (algorithms), not for codebases. What is the cyclomatic complexity of Facebook? :)
Probably at least 5
Re: A search engine in 80 lines of Python
#93Earlier quoted context omitted.
what software did you use for your own feed?
Various scripts in python, shell and javascript [0]. [0] https://github.com/abetusk/www.mechaelephant.com/tree/releas...
thanks! I'm currently looking at the files
how did you filter which posts appear in the feed?
EDIT: I was able to make it work, I replaced the node module xmltojson with xq (python lib available from pip)
Re: A search engine in 80 lines of Python
#94Nice! It wouldn't be much work to add fuzzy-search functionality (all results with a prefix edit-distance below some threshold delta, so that a search for "hackrnew" matches "hackernews"). Basically, what you would do is you add an additional inverted index, but this time the keys are n-grams of words in your document collection (typically 3-grams), and the postings are the words (or their ID) in which these n-grams…
How much does that increase the index size?
Re: A search engine in 80 lines of Python
#95Looking at the code (src/microsearch/engine.py), we have: class SearchEngine: def __init__(self, k1: float = 1.5, b: float = 0.75): self._index: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._documents: dict[str, str] = {} self.k1 = k1 self.b = b I've no idea what `k1` or `b` are. Nor is there a single comment in the entire file. Are comments considered unfashionable these days? Looking at `_…
Hi, author here. If I wanted a catchy title for the post I needed to cut the number of LOC as much as possible;) Joking apart, thanks for your feedback. I agree that usually it's better to have documentation and code together, but in this case since it's an educational project I decided to split code and documentation, and document the code in a blog post.
Put the comments at the end of the lines at least, maybe? LOC remains the same, but now there are at least some comments.
Re: A search engine in 80 lines of Python
#96Looking at the code (src/microsearch/engine.py), we have: class SearchEngine: def __init__(self, k1: float = 1.5, b: float = 0.75): self._index: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._documents: dict[str, str] = {} self.k1 = k1 self.b = b I've no idea what `k1` or `b` are. Nor is there a single comment in the entire file. Are comments considered unfashionable these days? Looking at `_…
That's explained in the article, which serves as the documentation for the code within the article. The link for BM25 goes to some of the math, and a little more searching the internet about BM25 parameters can lead you to some relevant articles on how to choose them.
BM25 is just too famous in traditional searching and not worth explaining it all over again.
Re: A search engine in 80 lines of Python
#97This is very cool and very educational. Don't deploy it, though. :-) I needed something like this once, but at a little bit larger scale (few tens of thousands of documents). The answer, as always, was [sqlite]( https://www.sqlite.org/fts5.html); structurally though it's just what you have here but with someone else writing the inverted-index persistence layer.
Wow it literally has the same equations and everything. Thanks for this comment, gave me a "shiver of understanding" if that makes sense haha.
Re: A search engine in 80 lines of Python
#98just fyi, PageRank and BM25 address orthogonal problems of reputation/popularity (PageRank) and query/semantic matching (BM25). Saying you're using bm25 instead of PageRank makes no sense.
both are similar in the sense that they can be used to sort a series of sites/documents by relevance. PageRank makes the assumption that sites with better in/out links are more relevant, while BM25 makes the assumption that documents with high keyword matching are more relevant. In this sense both solutions are similar.
Re: A search engine in 80 lines of Python
#99Earlier quoted context omitted.
what software did you use for your own feed?
Various scripts in python, shell and javascript [0]. [0] https://github.com/abetusk/www.mechaelephant.com/tree/releas...
https://static.nani-so.re/feed/ many thanks!