Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a full-text search engine in 150 lines of Python code

bart.degoe.de

Post reply on HN
Print view View original

31–40 of 88 posts

Re: Building a full-text search engine in 150 lines of Python code

#31

Post by throwaway320912 »

machiaweliczny wrote: post #29

Cool article. I recently build program in Go that takes wikipedia article and gets all dependencies then using tfidf*count ranks concepts in order of "importance". Seems quite good for math articles to get list of more basic concepts to understand first.

Fun times. I once applied pagerank onto a set of 8000 math articles and ran the result as a web app in 2009/10.

http://web.archive.org/web/20091230103939/http://myyn.org/m

As a gimmick, I created 36 groups, with group 1 containing the most important concepts:

http://web.archive.org/web/20100109055506/http://myyn.org/m/...

Spoiler, the top ten concepts were:

Function · Set · Number · Integer · Real Number · Point · Property · Finite · Ring · Relation Theory

BTW: Glad the archive has a copy, because I do not.

Re: Building a full-text search engine in 150 lines of Python code

#35

Post by johnwheeler »

diogenesjunior wrote: post #34
KAdot wrote: post #33

The article looks suspiciously similar to https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full... . Very similar examples, code and structure.

Except that one is written in go and the other in python...

Yes, but where's the attribution?

Re: Building a full-text search engine in 150 lines of Python code

#36

Post by pmiller2 »

KAdot wrote: post #33

The article looks suspiciously similar to https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full... . Very similar examples, code and structure.

So? Wikipedia is one of the most convenient, large English corpora available, and I doubt there are many significantly different ways to write the bit of functionality that's built up here. I'm not sure if that's what you're meaning to suggest, or that there was some kind of plagiarism / inspiration going on here.

Re: Building a full-text search engine in 150 lines of Python code

#37

Post by pmiller2 »

johnwheeler wrote: post #35
diogenesjunior wrote: post #34

Earlier quoted context omitted.

Except that one is written in go and the other in python...

Yes, but where's the attribution?

How many different ways do you think there are to write this code?

Re: Building a full-text search engine in 150 lines of Python code

#38

Post by rakoo »

habibur wrote: post #28
rakoo wrote: post #15

Earlier quoted context omitted.

SQLite has a pretty good built-in fts engine: https://www.sqlite.org/fts5.html

Problem is FTS5 isn't included in the most default installation through package managers [I use Fedora]. And recompiling from source breaks a lot of things, as sqlite libraries are generally linked with all apps that use it.

I admit I only used sqlite through the go driver (https://github.com/mattn/go-sqlite3) where using fts5 amounts to one flag during the compile phase.

Re: Building a full-text search engine in 150 lines of Python code

#40

Post by asdfasgasdgasdg »

ignoramous wrote: post #19

Reminds of this David Crawshaw (CTO at Tailscale) presentation on full-text search with SQLite which probably requires 10 lines or less: https://www.youtube-nocookie.com/embed/RqubKSF3wig

It's not relevant or an indicator of power that this example requires ten lines of SQL. This presentation just uses SQLite's builtin full-text search system [1]. Of course it's going to require less code to call a library than to implement it.

[1]: https://sqlite.org/fts5.html

Post reply on HN