Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a full-text search engine in 150 lines of Python code

bart.degoe.de

Post reply on HN
Print view View original

1–10 of 88 posts

Re: Building a full-text search engine in 150 lines of Python code

#4

Post by mlthoughts2018 »

I worked previously for a very high traffic ecommerce company (Alexa top 300 site).

As part of the search team, I worked on a project where we deliberately rewrote the whole product search engine in Python and Cython, including our own algorithms manipulating documents for deletion, low latency reindexing after edits, and more.

We did this because SOLR was too slow and the process of defining custom sort orders (for example, new sort orders defined by machine learning ranking algorithms, and needing to be A/B tested regularly) was awful and performance in SOLR was poor.

It was a really fun project. One of the slogans for our group at the time was “rewriting search in Python for speed.”

The ultimate system we deployed was insanely fast. I doubt you could have made it faster even writing the entire thing directly in C or C++. It became a reference project in the company to help avoid various flavors of arrogant dismissal of Python as a backend service language for performance critical systems.

Re: Building a full-text search engine in 150 lines of Python code

#5

Post by NicoJuicy »

mlthoughts2018 wrote: post #4

I worked previously for a very high traffic ecommerce company (Alexa top 300 site). As part of the search team, I worked on a project where we deliberately rewrote the whole product search engine in Python and Cython, including our own algorithms manipulating documents for deletion, low latency reindexing after edits, and more. We did this because SOLR was too slow and the process of defining custom sort orders (for…

Could i send you a mail somewhere? I'd like to have a casual conversation about e-commerce and some of your insights.

Re: Building a full-text search engine in 150 lines of Python code

#6

Post by bartdegoede »

habibur wrote: post #2

Excellent read. Was searching for a full text search engine but not finding any suitable one. Plan to implement one just this way.

Author here: you may want to check out something like Whoosh https://whoosh.readthedocs.io/en/latest/intro.html (it's like a clone of Lucene but in pure Python). I've used this to build some basic search for a small Python website and it was more than fast enough for my purposes :-)
Post reply on HN