A well worked example using Python and Redis:
https://redislabs.com/ebook/part-2-core-concepts/chapter-7-s...
Building a full-text search engine in 150 lines of Python code
71–80 of 88 posts
Re: Building a full-text search engine in 150 lines of Python code
#72Failed for me. I use Chinese and Japanese. There are no spaces to split on. I also search code where this also fails. I know it was meant to be simple and illustrate some points. I think the point it fails at is that this is actually a much harder problem in real life than a simple 150 line solution suggests.
if you use a library for Chinese/Japanese tokenization (which is harder because the lack of space), it seems like the rest of the code would work?
Re: Building a full-text search engine in 150 lines of Python code
#73Earlier quoted context omitted.
it amaze me how much is about creating an eco-system to scale and handling complexity and how software grow to answer that instead of trying to split into two different piece of software. Many people/companies/websites just need those basic features when search is not a core essential capability
Well, Solr and Lucene projects are right now in the process of splitting up (after joining for versions 3-8). And, you could have always used Lucene directly as an embedded search library. Amazon does, for example, for their customer facing search as their scalability patterns do not align with either Solr or Elasticsearch. And if you do use Lucene directly, you can choose just the libraries that apply to your use ca…
> Amazon does, for example, for their customer facing search as their scalability patterns do not align with either Solr or Elasticsearch.
Can you explain this to a non-programmer?
Re: Building a full-text search engine in 150 lines of Python code
#74Earlier quoted context omitted.
>Many people/companies/websites just need those basic features when search is not a core essential capability and some companies convince themselves that search isn't essential to them and they end up with awful search not much better than an SQL like query and when they do UX evaluations of how people use their site say look - nobody uses our crappy search let's not put any time into fixing it!
agreed. but lets be honest to operate an elasticsearch for instance. You need a sizeable team. its not a one man job.
well, I'm pretty good at all the non operations aspects of ElasticSearch, so in my experience if you have ES in some sort of managed cloud instance - like searchly, doesn't have to be AWS or elastic - it's a one man job - and also in my experience not necessarily full time devoted to just es.
May vary dependent on if you have a lot of data, but if so you probably have people to handle lots of data problems anyway.
Re: Building a full-text search engine in 150 lines of Python code
#75Excellent read. Was searching for a full text search engine but not finding any suitable one. Plan to implement one just this way.
Many years ago I ran into this paper "Self-indexing inverted files for fast text retrieval" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.18....
It's short and to the point. And then I implemented all that ... in PHP and MySQL :)
It feels daunting at first, but once you understand what it wants you to do, it's actually not that hard (for this particular paper, and this particular approach).
However, you do want to employ a stemming library to normalize word forms.
Re: Building a full-text search engine in 150 lines of Python code
#76I went down this rabbit hole some ten years ago, and somewhere along the line I discovered SQLite has an FTS engine (up to version 5 now, I think) and just started using that instead. Massive performance for tens of gigs’ worth of text content on a single core.
Re: Building a full-text search engine in 150 lines of Python code
#77The article looks suspiciously similar to https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full... . Very similar examples, code and structure.
Does it? Both are implementations and explanations of a well-known algorithm. Most articles on quicksort will also look alike, but there's no reason to assume the author has plagiarized anything.
[deleted]
Re: Building a full-text search engine in 150 lines of Python code
#78Cool article. I recently build program in Go that takes wikipedia article and gets all dependencies then using tfidf*count ranks concepts in order of "importance". Seems quite good for math articles to get list of more basic concepts to understand first.
Fun times. I once applied pagerank onto a set of 8000 math articles and ran the result as a web app in 2009/10. http://web.archive.org/web/20091230103939/http://myyn.org/m As a gimmick, I created 36 groups, with group 1 containing the most important concepts: http://web.archive.org/web/20100109055506/http://myyn.org/m/... Spoiler, the top ten concepts were: Function · Set · Number · Integer · Real Number · Point · Pr…
Wow thats such a cool idea. Really nice.
Re: Building a full-text search engine in 150 lines of Python code
#79As Joel spolsky said: "Just do me a favor and search the damned hard drive, quickly, for the string I typed, using full-text indexes and other technologies that were boring in 1973."
That's a standard feature I use nearly everyday and I'm very glad for in MacOS.