Does anybody work for Atlassian here? If so, can you please share this with the Confluence team? Thanks...
Do people really find the search in confluence that bad? Would they actually be willing to pay to have it fixed?
Building a full-text search engine in 150 lines of Python code
21–30 of 88 posts
Re: Building a full-text search engine in 150 lines of Python code
#22Does anybody work for Atlassian here? If so, can you please share this with the Confluence team? Thanks...
If you want to feel extra sad about it, see this: https://jira.atlassian.com/browse/CONFSERVER-13499
If a wiki page has the word adrecordset, a search on recordset should include the page. Currently only a search on adrecordset or a search with wildcards returns the page.
Wait..this seems unreasonable?
Isn't this how it works with elasticsearch or solr? or even google for that matter..a search for green won't return evergreen..
Re: Building a full-text search engine in 150 lines of Python code
#23Earlier quoted context omitted.
If you want to feel extra sad about it, see this: https://jira.atlassian.com/browse/CONFSERVER-13499
If a wiki page has the word adrecordset, a search on recordset should include the page. Currently only a search on adrecordset or a search with wildcards returns the page. Wait..this seems unreasonable? Isn't this how it works with elasticsearch or solr? or even google for that matter..a search for green won't return evergreen..
They don't support leading wildcards. So foo* will find foobar but *bar won't find foobar. Elastic does fine with that as of 7.9.
Re: Building a full-text search engine in 150 lines of Python code
#24Using nlp techniques to create tokens is for a fast and simple python search often not needed. It makes things slow and python standard string function are often good enough. Issues arise when searching for combinations of words like ‘machine learning’ in a sentence. Nice read, but the GitHub repro needs a license to be more useful.
Re: Building a full-text search engine in 150 lines of Python code
#25Why would one choose Python for this instead of Go or Rust? I imagine Python performance would be terrible.
The idea was to illustrate the concepts of an inverted index and tf-idf :-) I've built some stuff like this for a smaller project written in Python because it was Easier™ than spinning up an Elasticsearch cluster (ie it definitely didn't need to scale :-D)
Yes, I've been really interested in FTS recently and love articles like this. I'm currently implementing a paired-down version myself in Elixir because I'm searching one "table" (not postgres) and I don't want to bring in external dependencies for it.
Re: Building a full-text search engine in 150 lines of Python code
#26I worked previously for a very high traffic ecommerce company (Alexa top 300 site). As part of the search team, I worked on a project where we deliberately rewrote the whole product search engine in Python and Cython, including our own algorithms manipulating documents for deletion, low latency reindexing after edits, and more. We did this because SOLR was too slow and the process of defining custom sort orders (for…
Defining custom sort orders in Solr is as simple as uploading a text file with the values you intend to use for ranking. This is a great feature that is in fact missing from Elasticsearch and saves you so much reindexing time. There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish.
> There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish.
I read that as a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. I am surprised that you read it that way. Who in their right mind would say Python speed is faster than C speed?
Re: Building a full-text search engine in 150 lines of Python code
#27Neat! One production ready python library I love to use in this space is https://radimrehurek.com/gensim/ It is quite mature and can handle a good amount of data well!
It offers topic modeling too and can b helpful to find similar documents also.
Re: Building a full-text search engine in 150 lines of Python code
#28Excellent read. Was searching for a full text search engine but not finding any suitable one. Plan to implement one just this way.
SQLite has a pretty good built-in fts engine: https://www.sqlite.org/fts5.html
Problem is FTS5 isn't included in the most default installation through package managers [I use Fedora]. And recompiling from source breaks a lot of things, as sqlite libraries are generally linked with all apps that use it.
Re: Building a full-text search engine in 150 lines of Python code
#29Cool article.
I recently build program in Go that takes wikipedia article and gets all dependencies then using tfidf*count ranks concepts in order of "importance". Seems quite good for math articles to get list of more basic concepts to understand first.
Re: Building a full-text search engine in 150 lines of Python code
#30Why would one choose Python for this instead of Go or Rust? I imagine Python performance would be terrible.
As a lot of the work is done by code library code, there's a very good chance that the code is not all that slow.