Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pg_bm25: Elastic-Quality Full Text Search Inside Postgres

docs.paradedb.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres

#61

Post by westurner »

arecurrence wrote: post #59
samokhvalov wrote: post #22

I checked the benchmarks and was surprised to see that native search is (a) so slow (seconds), and (b) demonstrating O(N) behavior – with indexing, it should not happen at all. Indeed, looking at the benchmark source code (thanks for providing it!), it completely lacks index for the native case, leading to a false statement the that native full-text search indexes Postgres provides (usually GIN indexes on tsvector co…

This sort of thing is more common with postgres than you'd think. I interviewed a candidate once whose company completely replaced querying in their postgres with elasticsearch because they could not figure out how to speed up certain text search queries. Nothing they tried would use the index.

"Preferred Index Types for Text Search" https://www.postgresql.org/docs/current/textsearch-indexes.h... :

> There are two kinds of indexes that can be used to speed up full text searches: GIN and GiST. Note that indexes are not mandatory for full text searching, but in cases where a column is searched on a regular basis, an index is usually desirable.

Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres

#63

Post by raybb »

retakeming wrote: post #26

Blog post author and one of the pg_bm25 contributors here. Super excited to see the interest in pg_bm25! pg_bm25 is our first step in building an Elasticsearch alternative on Postgres. We built it as a result of working on hybrid search in Postgres and becoming frustrated with Postgres' sparse feature set when it comes to full text search. To address a few of the discussion points, today pg_bm25 can be installed on s…

Could this also work as an alternative to Apache Solr? If so might be worth while to market it that way a bit.

I don't really know much about Solr but just started using it while helping with a project for openlibrary.org and it seems pretty alright but I'm still not totally sure I understand what makes it popular.

Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres

#64

Post by PSeitz »

fiedzia wrote: post #49
benpacker wrote: post #35

Earlier quoted context omitted.

The underlying engine, Tantivy, has better performance characteristics than Lucene. You can compare Lucene to Tantivy and can compare Elasticsearch to pg_bm25 or ParadeDB

It's faster, but misses tons of features, starting with geosearch. Hopefully they will come with wider use.

The issue for geo search is here: https://github.com/quickwit-oss/tantivy/issues/44

Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres

#65

Post by philippemnoel »

olivermuty wrote: post #54
philippemnoel wrote: post #42

Earlier quoted context omitted.

Our goal is for one day ParadeDB to be a viable alternative to AWS RDS/Aurora, so that like you say, you don't need to keep data in-sync and can just use one system (ParadeDB). Soon it will be possible for you to have ParadeDB running on your AWS (utilizing your cloud credits+all security/privacy guarantees) but be managed via the ParadeDB dashboard, similar to how Aurora works from a developer UX. Of course if you a…

Will you be providing this for bring-your-own-compute in general? There is a gaping hole in the market for this. All the big vendors that provide postgres as a service require you to be on very specific types of hosting like aws fargate, google gke etc (looking at you Crunchydata). We are using Scaleway (french cloud) which is heaven when it comes to GDPR and Schrems compliance, but once we grow out of their managed…

Hey! Absolutely, we would love to offer as many cloud providers as possible for our compute backend. We're starting with AWS, and will be adding other clouds based on demand. I've added Scaleway to our list, and if you'd like to help us bring ParadeDB to Scaleway we would love to work together to make it happen faster.

In the meantime, you can self-host ParadeDB on Scaleway directly by running the Docker container. Hope this helps!

Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres

#67

Post by rdhatt »

raybb wrote: post #63
retakeming wrote: post #26

Blog post author and one of the pg_bm25 contributors here. Super excited to see the interest in pg_bm25! pg_bm25 is our first step in building an Elasticsearch alternative on Postgres. We built it as a result of working on hybrid search in Postgres and becoming frustrated with Postgres' sparse feature set when it comes to full text search. To address a few of the discussion points, today pg_bm25 can be installed on s…

Could this also work as an alternative to Apache Solr? If so might be worth while to market it that way a bit. I don't really know much about Solr but just started using it while helping with a project for openlibrary.org and it seems pretty alright but I'm still not totally sure I understand what makes it popular.

Solr and Elasticsearch are both Java servers built on top of the Java search library Lucene. There are plenty of articles on the internet describing how they differ. However since they share the same core, so they are very similar as well. For the context of this discussion, you can consider Solr & Elasticsearch as interchangeable - a potayto, potahto situation.

Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres

#68

Post by machty »

What kind of "consistency" do bm25 indexes offer? e.g. I think ElasticSearch is eventually consistent and is constantly indexing in the background and classic Postgres GIN indexes have configuration like `gin_pending_list_limit` and `fastupdate` functionality to avoid slowdowns on insertions (and then you get slowdowns when an insert hits the threshold and triggers the catch-up indexing).

Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres

#69

Post by philippemnoel »

machty wrote: post #68

What kind of "consistency" do bm25 indexes offer? e.g. I think ElasticSearch is eventually consistent and is constantly indexing in the background and classic Postgres GIN indexes have configuration like `gin_pending_list_limit` and `fastupdate` functionality to avoid slowdowns on insertions (and then you get slowdowns when an insert hits the threshold and triggers the catch-up indexing).

ParadeDB and pg_bm25 offer weak consistency. pg_bm25 doesn't slow down transactions for indexing, and like ElasticSearch it becomes become eventually consistent shortly after (typically at most a few seconds, altough your mileage may vary based on the amount of data modified in the transaction(s)).

Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres

#70

Post by j45 »

philippemnoel wrote: post #28
j45 wrote: post #24

Earlier quoted context omitted.

Running it for your own purposes as part of a solution that includes search should be fine under AGPL. If your product is elastic search built into Postgres as a repackaged and direct competitor to this search plug-in, that’s where my understanding is over the line.

ParadeDB author here -- correct! We plan to offer a hosted version soon and the idea behind picking AGPL is to be as permissive as possible so that people can use the product for free, but also protect ourselves from abuse in case a large company, say AWS, were to want to ship it in their own environment. In fact, we went through much questioning wondering to go with ELv2, Apache, AGPL, etc. before settling on AGPL

Appreciate the response! This would be a great blog post btw.
Post reply on HN