Earlier quoted context omitted.
Yes, but where's the attribution?
How many different ways do you think there are to write this code?
A billion?
Building a full-text search engine in 150 lines of Python code
51–60 of 88 posts
Re: Building a full-text search engine in 150 lines of Python code
#52I worked previously for a very high traffic ecommerce company (Alexa top 300 site). As part of the search team, I worked on a project where we deliberately rewrote the whole product search engine in Python and Cython, including our own algorithms manipulating documents for deletion, low latency reindexing after edits, and more. We did this because SOLR was too slow and the process of defining custom sort orders (for…
Defining custom sort orders in Solr is as simple as uploading a text file with the values you intend to use for ranking. This is a great feature that is in fact missing from Elasticsearch and saves you so much reindexing time. There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish.
You are incorrect. This is only true if you can precompute all sort orders (many types of sort orders cannot be precomputed and depend on additional context data only available at query time, especially for personalization or trending solutions). Additionally you must deal with precommitment to very poor sharding properties with Solr. With our Python approach, we could hold all results (billions of content items with trimmed down data structures representing only the thinnest container needed for each sort order representation) in memory easily, and dynamically resort on the fly or double sort by multiple sort orders that each required contextual data only available at query time.
Re: Building a full-text search engine in 150 lines of Python code
#53Earlier quoted context omitted.
> There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish. I read that as a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. I am surprised that you read it that way. Who in their right mind would say Python speed is faster than C…
You read >I doubt you could have made it faster even writing the entire thing directly in C or C++. as > a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. ?
You seem fundamentally confused, for example with tools like Cython.
Many extension module implementations in Python are literally as fast as pure C (not just nearly as fast with minor extra CPython overhead, but literally as fast as pure C by deliberately bypassing CPython VM loop and data models).
Re: Building a full-text search engine in 150 lines of Python code
#54 return [token for token in tokens if token]
Re: Building a full-text search engine in 150 lines of Python code
#55return [token for token in tokens if token] What the what?
Aka
list(filter(None, tokens))
list(filter(lambda x: bool(x), tokens))Re: Building a full-text search engine in 150 lines of Python code
#56Re: Building a full-text search engine in 150 lines of Python code
#57return [token for token in tokens if token] What the what?
Lol. Compact map.
Re: Building a full-text search engine in 150 lines of Python code
#58return [token for token in tokens if token] What the what?
Comprehensions in Python are pretty handy, but they can look weird sometimes. This is filtering out `None`, empty string, etc values from a list.
It is iterating over a list (tokens) and creating a temporary variable (token). It tests the truthyness of it (if token), which means None and '' will return False and thus be excluded, and then returns it (the first token).
Re: Building a full-text search engine in 150 lines of Python code
#59The article looks suspiciously similar to https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full... . Very similar examples, code and structure.
So? Wikipedia is one of the most convenient, large English corpora available, and I doubt there are many significantly different ways to write the bit of functionality that's built up here. I'm not sure if that's what you're meaning to suggest, or that there was some kind of plagiarism / inspiration going on here.
I think it is plagiarize. Compare "def analyze" and "func analyze". Very similar.
Re: Building a full-text search engine in 150 lines of Python code
#60The article looks suspiciously similar to https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full... . Very similar examples, code and structure.
Does it? Both are implementations and explanations of a well-known algorithm. Most articles on quicksort will also look alike, but there's no reason to assume the author has plagiarized anything.
Compare "def analyze" and "func analyze". It's obvious this article was copying the other.