Failed for me. I use Chinese and Japanese. There are no spaces to split on. I also search code where this also fails. I know it was meant to be simple and illustrate some points. I think the point it fails at is that this is actually a much harder problem in real life than a simple 150 line solution suggests.
Building a full-text search engine in 150 lines of Python code
61–70 of 88 posts
Re: Building a full-text search engine in 150 lines of Python code
#62Re: Building a full-text search engine in 150 lines of Python code
#63Re: Building a full-text search engine in 150 lines of Python code
#64Nice introduction. A good ramp-up from basic splitting to ranking. It does need to be said that when Lucene had a set of features this small, it was also pretty tiny. And, if those are the needs, one could still download it and it will probably run on modern JVM: https://archive.apache.org/dist/lucene/java/ lucene-1.4.3.jar 2004-11-29 14:13 316K It is a bit bigger of course, but that's because it already had stemmers…
it amaze me how much is about creating an eco-system to scale and handling complexity and how software grow to answer that instead of trying to split into two different piece of software. Many people/companies/websites just need those basic features when search is not a core essential capability
Re: Building a full-text search engine in 150 lines of Python code
#65Re: Building a full-text search engine in 150 lines of Python code
#66Nice introduction. A good ramp-up from basic splitting to ranking. It does need to be said that when Lucene had a set of features this small, it was also pretty tiny. And, if those are the needs, one could still download it and it will probably run on modern JVM: https://archive.apache.org/dist/lucene/java/ lucene-1.4.3.jar 2004-11-29 14:13 316K It is a bit bigger of course, but that's because it already had stemmers…
it amaze me how much is about creating an eco-system to scale and handling complexity and how software grow to answer that instead of trying to split into two different piece of software. Many people/companies/websites just need those basic features when search is not a core essential capability
Well, Solr and Lucene projects are right now in the process of splitting up (after joining for versions 3-8).
And, you could have always used Lucene directly as an embedded search library. Amazon does, for example, for their customer facing search as their scalability patterns do not align with either Solr or Elasticsearch.
And if you do use Lucene directly, you can choose just the libraries that apply to your use case. I think at minimum, you can get away with maybe 3 jars (core, queryparser, analyzers-common) and that's just over 5Mb for the latest Lucene, which includes things like: http://blog.mikemccandless.com/2021/03/open-source-collabora...
Sometimes, there is a disconnect between implementation that is very flexible and messaging that just shows 'use everything' garden path.
Re: Building a full-text search engine in 150 lines of Python code
#67The article looks suspiciously similar to https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full... . Very similar examples, code and structure.
I saw a case once, I forget where, when someone was accused of plagiarising from wikipedia and the truth was that they'd themselves written a large part of the wikipedia article.
Re: Building a full-text search engine in 150 lines of Python code
#68Nice introduction. A good ramp-up from basic splitting to ranking. It does need to be said that when Lucene had a set of features this small, it was also pretty tiny. And, if those are the needs, one could still download it and it will probably run on modern JVM: https://archive.apache.org/dist/lucene/java/ lucene-1.4.3.jar 2004-11-29 14:13 316K It is a bit bigger of course, but that's because it already had stemmers…
it amaze me how much is about creating an eco-system to scale and handling complexity and how software grow to answer that instead of trying to split into two different piece of software. Many people/companies/websites just need those basic features when search is not a core essential capability
>Many people/companies/websites just need those basic features when search is not a core essential capability
and some companies convince themselves that search isn't essential to them and they end up with awful search not much better than an SQL like query and when they do UX evaluations of how people use their site say look - nobody uses our crappy search let's not put any time into fixing it!
Re: Building a full-text search engine in 150 lines of Python code
#69Earlier quoted context omitted.
it amaze me how much is about creating an eco-system to scale and handling complexity and how software grow to answer that instead of trying to split into two different piece of software. Many people/companies/websites just need those basic features when search is not a core essential capability
>Many people/companies/websites just need those basic features when search is not a core essential capability and some companies convince themselves that search isn't essential to them and they end up with awful search not much better than an SQL like query and when they do UX evaluations of how people use their site say look - nobody uses our crappy search let's not put any time into fixing it!
agreed. but lets be honest to operate an elasticsearch for instance. You need a sizeable team. its not a one man job.
Re: Building a full-text search engine in 150 lines of Python code
#70Earlier quoted context omitted.
> There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish. I read that as a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. I am surprised that you read it that way. Who in their right mind would say Python speed is faster than C…
You read >I doubt you could have made it faster even writing the entire thing directly in C or C++. as > a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. ?
[deleted]