Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizing Top K in Postgres

paradedb.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Optimizing Top K in Postgres

#21

Post by philippemnoel »

bob1029 wrote: post #9

Lucene really does feel like magic sometimes. It was designed expressly to solve the top K problem at hyper scale. It's incredibly mature technology. You can go from zero to a billion documents without thinking too much about anything other than the amount of mass storage you have available. Every time I've used Lucene I have combined it with a SQL provider. It's not necessarily about one or the other. The FTS facili…

Under the hood, ParadeDB is built by integrating Tantivy (a Lucene-inspired Rust search library) inside Postgres. This is to say: I agree with you -- We're not trying to claim that Postgres itself is an alternative to Lucene, but rather that something like Lucene can be integrated inside Postgres so that you can get the power of both in a single system (or in a cluster of Postgres instances)

Re: Optimizing Top K in Postgres

#23

Post by monadicmonad »

philippemnoel wrote: post #21
bob1029 wrote: post #9

Lucene really does feel like magic sometimes. It was designed expressly to solve the top K problem at hyper scale. It's incredibly mature technology. You can go from zero to a billion documents without thinking too much about anything other than the amount of mass storage you have available. Every time I've used Lucene I have combined it with a SQL provider. It's not necessarily about one or the other. The FTS facili…

Under the hood, ParadeDB is built by integrating Tantivy (a Lucene-inspired Rust search library) inside Postgres. This is to say: I agree with you -- We're not trying to claim that Postgres itself is an alternative to Lucene, but rather that something like Lucene can be integrated inside Postgres so that you can get the power of both in a single system (or in a cluster of Postgres instances)

why do I want them in a single system?

Re: Optimizing Top K in Postgres

#24

Post by ddorian43 »

monadicmonad wrote: post #23
philippemnoel wrote: post #21

Earlier quoted context omitted.

Under the hood, ParadeDB is built by integrating Tantivy (a Lucene-inspired Rust search library) inside Postgres. This is to say: I agree with you -- We're not trying to claim that Postgres itself is an alternative to Lucene, but rather that something like Lucene can be integrated inside Postgres so that you can get the power of both in a single system (or in a cluster of Postgres instances)

why do I want them in a single system?

Its easier to have 1 system until a certain scale.

Re: Optimizing Top K in Postgres

#25

Post by tacone »

hrmtst93837 wrote: post #16
tacone wrote: post #7

The issue here is the row based format. You simply can't filter on arbitrary columns with that. Either use an external warehouse or a columnar plug-in like Timescale.

[flagged]

My apologies, the article was talking about 100M-rows datasets so I presumed that was the scenario.

Re: Optimizing Top K in Postgres

#26

Post by inbx0 »

monadicmonad wrote: post #23
philippemnoel wrote: post #21

Earlier quoted context omitted.

Under the hood, ParadeDB is built by integrating Tantivy (a Lucene-inspired Rust search library) inside Postgres. This is to say: I agree with you -- We're not trying to claim that Postgres itself is an alternative to Lucene, but rather that something like Lucene can be integrated inside Postgres so that you can get the power of both in a single system (or in a cluster of Postgres instances)

why do I want them in a single system?

  - saves infra costs 
  - saves infra headaches 
  - devs only need to be experts in one system (or well I guess one and a half, probably there's something to learn about ParadeDB too, but probably less than in learning Lucine) 
  - no need to worry about keeping data up to date in the separate seach system
  - all data is available when you want to do new queries that you handn't thought of when implementing the data transfer/indexing
Post reply on HN