Optimizing Top K in Postgres
paradedb.com
Optimizing Top K in Postgres
1–10 of 26 posts
Re: Optimizing Top K in Postgres
#2[flagged]
Re: Optimizing Top K in Postgres
#3The "But Wait, We Need Filters Too" paragraph mentions "US" filter which is introduced only later on.
Re: Optimizing Top K in Postgres
#4[dead]
Re: Optimizing Top K in Postgres
#5The "But Wait, We Need Filters Too" paragraph mentions "US" filter which is introduced only later on.
And footnote 3 is unreferenced.
Re: Optimizing Top K in Postgres
#6Maybe I'm wrong, but for this query:
SELECT * FROM benchmark_logs WHERE severity this index
CREATE INDEX ON benchmark_logs (severity, timestamp);
cannot be used as proposed: "Postgres can jump directly to the portion of the tree matching severity Postgres with this index can walk to a part of the tree with severity < 3, but timestamps are sorted only for the same severity.
Re: Optimizing Top K in Postgres
#7The issue here is the row based format. You simply can't filter on arbitrary columns with that. Either use an external warehouse or a columnar plug-in like Timescale.
Re: Optimizing Top K in Postgres
#8Maybe I'm wrong, but for this query: SELECT * FROM benchmark_logs WHERE severity this index CREATE INDEX ON benchmark_logs (severity, timestamp); cannot be used as proposed: "Postgres can jump directly to the portion of the tree matching severity Postgres with this index can walk to a part of the tree with severity < 3, but timestamps are sorted only for the same severity.
If severity is a low cardinality enum, it still seems acceptable
Re: Optimizing Top K in Postgres
#9Lucene really does feel like magic sometimes. It was designed expressly to solve the top K problem at hyper scale. It's incredibly mature technology. You can go from zero to a billion documents without thinking too
much about anything other than the amount of mass storage you have available.
Every time I've used Lucene I have combined it with a SQL provider. It's not necessarily about one or the other. The FTS facilities within the various SQL providers are convenient, but not as capable by comparison. I don't think mixing these into the same thing makes sense. They are two very different animals that are better joined by way of the document ids.
Re: Optimizing Top K in Postgres
#10Just in case, there is a btree_gin extension which can be used in queries combining gin-indexable column and btree-indexable column. It doesn’t solve top-K ordering problem though.