BM25 in PostgreSQL
blog.vectorchord.ai
BM25 in PostgreSQL
1–10 of 34 posts
Re: BM25 in PostgreSQL
#2Any comparison results in terms of performance vs accuracy with: https://github.com/paradedb/paradedb/tree/dev/pg_search
Re: BM25 in PostgreSQL
#3This looks very neat. In our company, for the past 10 years I've bet heavily on Postgres have not allowed any other databases to be used. Stuff like this makes me hope I can continue to do so as our database is growing.
But, it appears that we are hitting our limits at this point. We have a table with tens of millions of rows and the use case is that it's faceted search.
Something like Zillow, with tends of millions of listings and tens of columns that can be queried. I'm having a tough time scaling it up. Does anyone have experience building something like that in pg?
Re: BM25 in PostgreSQL
#4This looks very neat. In our company, for the past 10 years I've bet heavily on Postgres have not allowed any other databases to be used. Stuff like this makes me hope I can continue to do so as our database is growing. But, it appears that we are hitting our limits at this point. We have a table with tens of millions of rows and the use case is that it's faceted search. Something like Zillow, with tends of millions…
pg_bm25 - Optimized for faceted search & PGroonga - Full-text search with JSONB support
Re: BM25 in PostgreSQL
#5Any comparison results in terms of performance vs accuracy with: https://github.com/paradedb/paradedb/tree/dev/pg_search
Seems like they know about ParadeDB, but for some reason don't publish banchmarks:
"""Another solution is ParadeDB, which pushes full-text search queries down to Tantivy for results. It supports BM25 scoring and complex query patterns like negative terms, aiming to be a complete replacement for ElasticSearch. However, it uses its own unique syntax for filtering and querying and delegates filtering operations to Tantivy instead of relying on Postgres directly. Its implementation requires several hooks into Postgres' query planning and storage, potentially leading to compatibility issues."""
So it is more apples to red than equal comparison.
Re: BM25 in PostgreSQL
#6Re: BM25 in PostgreSQL
#7I'd be interested to see how this stacks up against Manticore or Meilisearch.
Re: BM25 in PostgreSQL
#8Can this be used on AWS RDS? I’ve seen a few things like this that would be great to use but without RDS support they’re unusable for us.
Re: BM25 in PostgreSQL
#9> Our next step is to fully decouple the tokenization process, transforming it into an independent and extensible extension. This will enable us to support multiple languages, allow users to customize tokenization for better results, and even incorporate advanced features like synonym handling.
This would be rad
Re: BM25 in PostgreSQL
#10This looks very neat. In our company, for the past 10 years I've bet heavily on Postgres have not allowed any other databases to be used. Stuff like this makes me hope I can continue to do so as our database is growing. But, it appears that we are hitting our limits at this point. We have a table with tens of millions of rows and the use case is that it's faceted search. Something like Zillow, with tends of millions…
pg_bm25 - Optimized for faceted search & PGroonga - Full-text search with JSONB support
Unrelated, but PGroonga reads funny in Argentinian. Better than gimp though