Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a full-text search engine in 150 lines of Python code

bart.degoe.de

Post reply on HN
Print view View original

11–20 of 88 posts

Re: Building a full-text search engine in 150 lines of Python code

#11

Post by inertiatic »

mlthoughts2018 wrote: post #4

I worked previously for a very high traffic ecommerce company (Alexa top 300 site). As part of the search team, I worked on a project where we deliberately rewrote the whole product search engine in Python and Cython, including our own algorithms manipulating documents for deletion, low latency reindexing after edits, and more. We did this because SOLR was too slow and the process of defining custom sort orders (for…

Defining custom sort orders in Solr is as simple as uploading a text file with the values you intend to use for ranking. This is a great feature that is in fact missing from Elasticsearch and saves you so much reindexing time.

There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish.

Re: Building a full-text search engine in 150 lines of Python code

#14

Post by bartdegoede »

creamytaco wrote: post #12

Why would one choose Python for this instead of Go or Rust? I imagine Python performance would be terrible.

The idea was to illustrate the concepts of an inverted index and tf-idf :-) I've built some stuff like this for a smaller project written in Python because it was Easier™ than spinning up an Elasticsearch cluster (ie it definitely didn't need to scale :-D)

Re: Building a full-text search engine in 150 lines of Python code

#17

Post by arafalov »

Nice introduction. A good ramp-up from basic splitting to ranking.

It does need to be said that when Lucene had a set of features this small, it was also pretty tiny. And, if those are the needs, one could still download it and it will probably run on modern JVM:

https://archive.apache.org/dist/lucene/java/

lucene-1.4.3.jar 2004-11-29 14:13 316K

It is a bit bigger of course, but that's because it already had stemmers, multilingual support, multiple Query Parsers including phrase, RAM and Disk directory implementations and a bunch of other advanced features one can easily see by unzipping the jar file.

Re: Building a full-text search engine in 150 lines of Python code

#20

Post by edoceo »

ignoramous wrote: post #19

Reminds of this David Crawshaw (CTO at Tailscale) presentation on full-text search with SQLite which probably requires 10 lines or less: https://www.youtube-nocookie.com/embed/RqubKSF3wig

Yea! FTS5 FTW! Amazing for what it costs.
Post reply on HN