Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a full-text search engine in 150 lines of Python code

bart.degoe.de

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Building a full-text search engine in 150 lines of Python code

#81

Post by arafalov »

fakedang wrote: post #73
arafalov wrote: post #66

Earlier quoted context omitted.

Well, Solr and Lucene projects are right now in the process of splitting up (after joining for versions 3-8). And, you could have always used Lucene directly as an embedded search library. Amazon does, for example, for their customer facing search as their scalability patterns do not align with either Solr or Elasticsearch. And if you do use Lucene directly, you can choose just the libraries that apply to your use ca…

> Amazon does, for example, for their customer facing search as their scalability patterns do not align with either Solr or Elasticsearch. Can you explain this to a non-programmer?

That is a good challenge, as I am sure their reasons were extremely technical. Twitter also uses Lucene and also in different ways. And I am not at Amazon or Twitter, so I just go by their public presentations, quite technical ones.

But basically I suspect it is like this.

Vast Majority of actual search magic (vector analysis and heavy math and optimized index management) is in Lucene. But Lucene does not care much about getting data into the system (json, csv, xml, etc) or getting it out. It just has internal representation of documents and fields. Plus it does not do any sort of multi-index management, needed for scale.

So, both Solr and Elasticsearch build on top of Lucene to do the user-friendly and scaling parts. Solr allows to send data in multiple formats, Elasticsearch sticks to JSON. Solr has multiple way to build search queries (url params, xml, json, changed over time), Elasticsearch sticks to JSON. Solr is a bit rigid about how schema (field collection and type definition) is done, Elastisearch is a bit more hands-off. Solr has composable pre-processor chains and explicit field analyzer chains, Elasticsearch focuses more on logs and has external pre-processors and custom scripting language. Solr uses Apache Zookeeper to coordinate distributed state, Elasticsearch rolled their own. Solr has one particular way to split data into replicas and shards and throw them from one server to another, Elasticsearch has another. Solr is perfectly happy to run a single node=server=collection=core. Elasticsearch starts with fully distributed cloud setup.

All of these are trade-offs on top of actual search. That's why most of the consultants work with both Solr and Elasticsearch, the most difficult search optimization concepts are same for both.

Bloomberg, I think, uses Solr directly and builds on top of that. They contributed Machine Learning ranking to Solr, for example. They also use Elasticsearch for log analysis, I believe.

Amazon's needs are different again. Their data comes in through different routes (direct from databases?), their multi-tier replication and sharding strategy have different needs, they don't need user-facing user interface, etc. So, they take Lucene directly, and build the rest on top. And then contribute back to Lucene, which makes it available to both Solr and Elasticsearch. Everybody benefits in the end.

Re: Building a full-text search engine in 150 lines of Python code

#82

Post by arafalov »

finikytou wrote: post #69
bryanrasmussen wrote: post #68

Earlier quoted context omitted.

>Many people/companies/websites just need those basic features when search is not a core essential capability and some companies convince themselves that search isn't essential to them and they end up with awful search not much better than an SQL like query and when they do UX evaluations of how people use their site say look - nobody uses our crappy search let's not put any time into fixing it!

agreed. but lets be honest to operate an elasticsearch for instance. You need a sizeable team. its not a one man job.

You can use embedded Lucene if you want great search but no UI/schema management and you are happy to code the integration. You can start with 3 jars (5Mb) in size and add additional jars as your needs expand (e.g. multilingual processing).

You can use a single node Solr, if you want UI and send data in multiple formats and may eventually need to scale.

You can use Elasticsearch if you know you are starting big and deal with scaling from the start.

You have the choice. It is not just clustered Elasticsearch vs code your own.

Re: Building a full-text search engine in 150 lines of Python code

#83

Post by arafalov »

ignoramous wrote: post #44
asdfasgasdgasdg wrote: post #40

Earlier quoted context omitted.

It's not relevant or an indicator of power that this example requires ten lines of SQL. This presentation just uses SQLite's builtin full-text search system [1]. Of course it's going to require less code to call a library than to implement it. [1]: https://sqlite.org/fts5.html

Relevant because "building a full-text search engine" is solved using SQLite just as much as it can be solved by writing Python on top of lxml and py-stemmer. SQLite is no heavyweight dependency àla Apache Lucene. It also helps that it is bundled in billions of Android and iOS devices.

Apache Lucene can be as small as 5Mb, if you don't load what you don't need.

Re: Building a full-text search engine in 150 lines of Python code

#84

Post by Bridgeburner4 »

@op great article. I ran your code first without fully reading the article. It took the code almost an hour to parse and index the data using the terminal.

A suggestion, if you want to make the code more friendly. Either:

1. Write a note to README.md that once you exit the program, you loose the indexed data.

or

2. Save the indices to disk. :)

thanks for the article

Re: Building a full-text search engine in 150 lines of Python code

#85

Post by IncRnd »

inertiatic wrote: post #45
IncRnd wrote: post #26

Earlier quoted context omitted.

> There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish. I read that as a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. I am surprised that you read it that way. Who in their right mind would say Python speed is faster than C…

You read >I doubt you could have made it faster even writing the entire thing directly in C or C++. as > a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. ?

Yes. If the implementation language isn't the determining factor for speed, then what is the cause? There is a branch of Computer Science called Algorithmics[0], wherein one expression of measurement is called Big-O notation[1].

[0] https://en.wikipedia.org/wiki/Algorithmics

[1] https://en.wikipedia.org/wiki/Big_O_notation

Re: Building a full-text search engine in 150 lines of Python code

#86

Post by boublepop »

skapadia wrote: post #56
argaba wrote: post #47

Earlier quoted context omitted.

Everything is a paraphrase of the original source - whatever and/or whenever that maybe from.

Including our very own DNA...

True, but it’s also common for people to acknowledge where their DNA came from.

You would likely be called out if you tried to hide that fact in order to seem more accomplished than your are. If for instance your dad is the owner of the company you are working at, you’d be rightfully be called out if you tried to hide the fact that your accomplishment is lessened by the fact that you copied your fathers DNA.

No one is claiming the author did anything wrong in copying the source material for his article. However it is wrong to not provide proper attribution, especially when it’s as easy as an “inspired by link” at the bottom.

Re: Building a full-text search engine in 150 lines of Python code

#87

Post by inertiatic »

mlthoughts2018 wrote: post #53
inertiatic wrote: post #45

Earlier quoted context omitted.

You read >I doubt you could have made it faster even writing the entire thing directly in C or C++. as > a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. ?

You seem fundamentally confused, for example with tools like Cython. Many extension module implementations in Python are literally as fast as pure C (not just nearly as fast with minor extra CPython overhead, but literally as fast as pure C by deliberately bypassing CPython VM loop and data models).

Cython can be as fast as C for simple use cases.

Because you're writing regular Python for a production service though, and not artificially writing optimized examples, then you will occasionally have to pay extra costs.

Are you perhaps a bit too invested in your own narrative?

Re: Building a full-text search engine in 150 lines of Python code

#88

Post by singhrac »

tarsinge wrote: post #79
happyweasel wrote: post #39

As Joel spolsky said: "Just do me a favor and search the damned hard drive, quickly, for the string I typed, using full-text indexes and other technologies that were boring in 1973."

That's a standard feature I use nearly everyday and I'm very glad for in MacOS.

Does Spotlight search even index contents? I didn't think that happened, but maybe I'm wrong?
Post reply on HN