In short: * Postgres still has the same problem with vacuum horizon, when a long-running query can block vacuuming of a quick-churning table. (The author uses a benchmark from 2015 when the problem was already well-understood.) * Stock Postgres still has no tools good enough against it. * The author's company special version of Postgres does have such tools; a few polite promotions of it are strewn across the article…
Keeping a Postgres Queue Healthy
31–34 of 34 posts
Re: Keeping a Postgres Queue Healthy
#32The problem is way worse if you update rows, if you stick with insert and delete you can get quite far.
0: https://www.cybertec-postgresql.com/en/is-update-the-same-as...
Re: Keeping a Postgres Queue Healthy
#33Earlier quoted context omitted.
Yes. For example you'll typically have a "budget" of 1-10k writes/sec. And a single heavy join can essentially take you offline. Even relatively modest enterprises typically need to shift some query patterns to OLAP/nosql/redis/etc. before very long.
can share our work setup we've been tinkering with at a mid size org. iceberg datalake + snowflake for our warehouse, iceberg tables live in s3, that is now shareable to postgres via the pg_lake extension which automagically context switches using duckdb under the hood to do olap queries acrossed the vast iceberg data. we keep the postgres db as an application db so apps can retrieve the broader data they want to sur…
Also, I am working on https://github.com/viggy28/streambed to perform analytics queries on S3 using DuckDB
Re: Keeping a Postgres Queue Healthy
#34Earlier quoted context omitted.
For disk usage, yes this doesn't address anything. But for read performance (which is IMO what the section in the article was motivated by), it doesn't actually matter to have a bunch of entries corresponding to dead tuples in your index, provided Postgres doesn't need to actually consider the dead tuples as part of your query. So if you have a monotonically increasing `job_id` and that's indexed, then so long as you…
The article is sparse on what pending means, but I would guess that that where condition would be enough?
Rough intuition: Postgres doesn't immediately delete rows, it just marks them as invalid after a certain snapshot version/transaction ID (and this mark is in the heap, not the indexes, AFAIK).
Every potential tuple that your index returns, Postgres needs to visit the heap to see if that tuple is alive. UNLESS all the tuples in that heap page are alive, in which case an optimisation called the 'visibility map' allows that check to skipped (relevant for Index-Only Scans, where Postgres can get all the results for your query from the index directly).
The only way to avoid the problem is therefore to either vacuum frequently enough that Postgres doesn't get any dead tuples returned from the index (that it must then visit the heap to check for liveness), or to bake in some condition to your queries that prevents the dead tuples from being returned from the index altogether. Vacuuming frequently is expensive and conflicts with having long-running transactions, so the latter option is generally the choice to go for when it matters.
[n.b. I feel I should note I am not a Postgres developer and wouldn't call myself an expert, just an enthusiast and dealt with a few problems here and there. So what I say should be taken with a grain or two of salt, though I believe it to be accurate.]