Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a full-text search engine in 150 lines of Python code

bart.degoe.de

Post reply on HN
Print view View original

41–50 of 88 posts

Re: Building a full-text search engine in 150 lines of Python code

#41

Post by denysvitali »

boyter wrote: post #21
denysvitali wrote: post #3

Does anybody work for Atlassian here? If so, can you please share this with the Confluence team? Thanks...

Do people really find the search in confluence that bad? Would they actually be willing to pay to have it fixed?

I wouldn't pay to fix it, Confluence isn't free. There are a lot of Open Source alternatives that provide such features out of the box. It's the only feature that shouldn't be broken.

I really hate Confluence nowadays (and I really liked in the past, when I was using it randomly): - Counter productive syntax that is not even equal to the one of Jira

- Search functionality that cannot find anything (I spend HOURS trying to find my content, let alone somebody else's). Some days I just wish I had the DB at hand / a folder with the whole pages exported to grep it myself.

- Proprietary file format - good luck migrating away from it

I'm really sad to say these things, I like Atlassian as a company but the more I use Atlassian products, the more I realize they're bloated and have _broken_ features that were implemented quickly just to tick some more boxes on their features sheet.

Re: Building a full-text search engine in 150 lines of Python code

#42

Post by denysvitali »

tyingq wrote: post #23
nova22033 wrote: post #22

Earlier quoted context omitted.

If a wiki page has the word adrecordset, a search on recordset should include the page. Currently only a search on adrecordset or a search with wildcards returns the page. Wait..this seems unreasonable? Isn't this how it works with elasticsearch or solr? or even google for that matter..a search for green won't return evergreen..

They don't support leading wildcards. So foo* will find foobar but *bar won't find foobar. Elastic does fine with that as of 7.9.

Well, in my experience they cannot even find URLs, IP addresses, or wildcarded words like foo*

Re: Building a full-text search engine in 150 lines of Python code

#43

Post by bartdegoede »

runningmike wrote: post #24

Using nlp techniques to create tokens is for a fast and simple python search often not needed. It makes things slow and python standard string function are often good enough. Issues arise when searching for combinations of words like ‘machine learning’ in a sentence. Nice read, but the GitHub repro needs a license to be more useful.

Added an MIT license. Have fun :-)

Re: Building a full-text search engine in 150 lines of Python code

#44

Post by ignoramous »

asdfasgasdgasdg wrote: post #40
ignoramous wrote: post #19

Reminds of this David Crawshaw (CTO at Tailscale) presentation on full-text search with SQLite which probably requires 10 lines or less: https://www.youtube-nocookie.com/embed/RqubKSF3wig

It's not relevant or an indicator of power that this example requires ten lines of SQL. This presentation just uses SQLite's builtin full-text search system [1]. Of course it's going to require less code to call a library than to implement it. [1]: https://sqlite.org/fts5.html

Relevant because "building a full-text search engine" is solved using SQLite just as much as it can be solved by writing Python on top of lxml and py-stemmer.

SQLite is no heavyweight dependency àla Apache Lucene. It also helps that it is bundled in billions of Android and iOS devices.

Re: Building a full-text search engine in 150 lines of Python code

#45

Post by inertiatic »

IncRnd wrote: post #26
inertiatic wrote: post #11

Earlier quoted context omitted.

Defining custom sort orders in Solr is as simple as uploading a text file with the values you intend to use for ranking. This is a great feature that is in fact missing from Elasticsearch and saves you so much reindexing time. There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish.

> There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish. I read that as a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. I am surprised that you read it that way. Who in their right mind would say Python speed is faster than C…

You read

>I doubt you could have made it faster even writing the entire thing directly in C or C++.

as

> a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C.

?

Re: Building a full-text search engine in 150 lines of Python code

#46

Post by jeroenhd »

KAdot wrote: post #33

The article looks suspiciously similar to https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full... . Very similar examples, code and structure.

Does it? Both are implementations and explanations of a well-known algorithm. Most articles on quicksort will also look alike, but there's no reason to assume the author has plagiarized anything.

Re: Building a full-text search engine in 150 lines of Python code

#47

Post by argaba »

KAdot wrote: post #33

The article looks suspiciously similar to https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full... . Very similar examples, code and structure.

Everything is a paraphrase of the original source - whatever and/or whenever that maybe from.

Re: Building a full-text search engine in 150 lines of Python code

#48

Post by asdfasgasdgasdg »

ignoramous wrote: post #44
asdfasgasdgasdg wrote: post #40

Earlier quoted context omitted.

It's not relevant or an indicator of power that this example requires ten lines of SQL. This presentation just uses SQLite's builtin full-text search system [1]. Of course it's going to require less code to call a library than to implement it. [1]: https://sqlite.org/fts5.html

Relevant because "building a full-text search engine" is solved using SQLite just as much as it can be solved by writing Python on top of lxml and py-stemmer. SQLite is no heavyweight dependency àla Apache Lucene. It also helps that it is bundled in billions of Android and iOS devices.

I'm sure there's some pypi package that has a full search system you can download. For that matter, there's the python sqlite package. It's irrelevant that it's ten lines of library calling code when you're comparing it to a toy implementation.
Post reply on HN