Seems really really cool. Is this a full DB, as in they have to take PG source, put in tantivy and their sauce, compile, and distribute? Or is this an extension? If it's the latter, what's the point of putting DB at the end of the name?
Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
41–50 of 72 posts
Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
#42Blog post author and one of the pg_bm25 contributors here. Super excited to see the interest in pg_bm25! pg_bm25 is our first step in building an Elasticsearch alternative on Postgres. We built it as a result of working on hybrid search in Postgres and becoming frustrated with Postgres' sparse feature set when it comes to full text search. To address a few of the discussion points, today pg_bm25 can be installed on s…
For what it's worth, the single biggest selling point to a better search, for me, would be not having to deal with additional infrastructure and all the hassle that comes with keeping data in sync. I would be very reluctant to move off of RDS/Aurora, and therefore have my principal motivation to use something like this is greatly negated. I understand that it becomes very hard to monetize if you're not able to offer…
Our goal is for one day ParadeDB to be a viable alternative to AWS RDS/Aurora, so that like you say, you don't need to keep data in-sync and can just use one system (ParadeDB). Soon it will be possible for you to have ParadeDB running on your AWS (utilizing your cloud credits+all security/privacy guarantees) but be managed via the ParadeDB dashboard, similar to how Aurora works from a developer UX.
Of course if you are 100% attached to AWS RDS itself (rather than the convenience of AWS RDS, which is replicable by ParadeDB), then there's not much we can do here, as we also need to eat :')
Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
#43Hey guys. Congratulations - this is an exciting development. Can you show some benchmarks around showing the count of matches -- `select count() from table where text match is there`?
This was the top reason that made us (Segmed.ai) give up on PostgreSQL FTS -- our folks require a very exact count of matches for medical conditions that are present in 20M reports. And doing COUNT() in PostgreSQL was crazy, crazy slow. If your extension could do simple len(invertedindex[word]) that would already be a great improvement.
ELK has it immediately, but at a cost of being one more thing to maintain, and the whole Logstash thing is clunky. I'd love to use FTS inside of PostgreSQL.
Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
#44Blog post author and one of the pg_bm25 contributors here. Super excited to see the interest in pg_bm25! pg_bm25 is our first step in building an Elasticsearch alternative on Postgres. We built it as a result of working on hybrid search in Postgres and becoming frustrated with Postgres' sparse feature set when it comes to full text search. To address a few of the discussion points, today pg_bm25 can be installed on s…
For what it's worth, the single biggest selling point to a better search, for me, would be not having to deal with additional infrastructure and all the hassle that comes with keeping data in sync. I would be very reluctant to move off of RDS/Aurora, and therefore have my principal motivation to use something like this is greatly negated. I understand that it becomes very hard to monetize if you're not able to offer…
Yes, I have a similar feeling towards Cloud SQL for Postgres. Would be great if Azure/GCP would be supported in some manner
Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
#45Blog post author and one of the pg_bm25 contributors here. Super excited to see the interest in pg_bm25! pg_bm25 is our first step in building an Elasticsearch alternative on Postgres. We built it as a result of working on hybrid search in Postgres and becoming frustrated with Postgres' sparse feature set when it comes to full text search. To address a few of the discussion points, today pg_bm25 can be installed on s…
For what it's worth, the single biggest selling point to a better search, for me, would be not having to deal with additional infrastructure and all the hassle that comes with keeping data in sync. I would be very reluctant to move off of RDS/Aurora, and therefore have my principal motivation to use something like this is greatly negated. I understand that it becomes very hard to monetize if you're not able to offer…
What are the features of RDS/Aurora that you need?
Also, it would be possible to set up a logical PG replica.
Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
#46Hey guys. Congratulations - this is an exciting development. Can you show some benchmarks around showing the count of matches -- `select count( ) from table where text match is there`? This was the top reason that made us (Segmed.ai) give up on PostgreSQL FTS -- our folks require a very exact count of matches for medical conditions that are present in 20M reports. And doing COUNT( ) in PostgreSQL was crazy, crazy slo…
I’m not sure if Postgres could support that type of operation directly via count() since I don’t know if the fact that no other filters are present is available to the Index Access Method API.
It might be possible to do a separate function though, like:
select pg_bm25_direct_count(‘term’)*
Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
#47Hey guys. Congratulations - this is an exciting development. Can you show some benchmarks around showing the count of matches -- `select count( ) from table where text match is there`? This was the top reason that made us (Segmed.ai) give up on PostgreSQL FTS -- our folks require a very exact count of matches for medical conditions that are present in 20M reports. And doing COUNT( ) in PostgreSQL was crazy, crazy slo…
I’m not sure if Postgres could support that type of operation directly via count() since I don’t know if the fact that no other filters are present is available to the Index Access Method API. It might be possible to do a separate function though, like: select pg_bm25_direct_count(‘term’)*
That would be fine--basically any way of achieving it would be fine. As of now, in PostgreSQL's FTS, I don't think there's any way to do this fast enough to give it back to the user.
Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
#48Earlier quoted context omitted.
For what it's worth, the single biggest selling point to a better search, for me, would be not having to deal with additional infrastructure and all the hassle that comes with keeping data in sync. I would be very reluctant to move off of RDS/Aurora, and therefore have my principal motivation to use something like this is greatly negated. I understand that it becomes very hard to monetize if you're not able to offer…
What are the features of RDS/Aurora that you need? Also, it would be possible to set up a logical PG replica.
Being in my VPC, having the support and track record of AWS, scaling to 128TB without me having to think about it, easy snapshots/backups.
Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
#49is this better than lucene
The underlying engine, Tantivy, has better performance characteristics than Lucene. You can compare Lucene to Tantivy and can compare Elasticsearch to pg_bm25 or ParadeDB
It's faster, but misses tons of features, starting with geosearch. Hopefully they will come with wider use.
Re: Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
#50I checked the benchmarks and was surprised to see that native search is (a) so slow (seconds), and (b) demonstrating O(N) behavior – with indexing, it should not happen at all. Indeed, looking at the benchmark source code (thanks for providing it!), it completely lacks index for the native case, leading to a false statement the that native full-text search indexes Postgres provides (usually GIN indexes on tsvector co…
I still can't figure out how pg_trgm is supposed to work for multi-term searches and how to ensure the dictionary table it needs stays up-to-date. Is there a good writeup somewhere?