Live data from Hacker News

PostgreSQL Parallel Query v2

rhaas.blogspot.com

21–30 of 44 posts

Re: PostgreSQL Parallel Query v2

#21

Earlier quoted context omitted.

Citus should already be saturating i/o during a scan unless you're only hitting one shard. What am I missing about your use case?

Doesn't this inherently depend on the query? Shard-local i/o saturation affects the latency of the entire query if you're not expecting the i/o to the client to saturate.

This was exactly my point. But usually you're doing a full scan across all shards, so disk i/o is being saturated by virtue of the MPP sharding.

Re: PostgreSQL Parallel Query v2

#22

Besides parallel queries (specifically parallel index scans), the Postgres 10 feature I'm most looking forward to is the work Andres Freund has done on using LLVM to JIT compile expressions[0]. Based on the the layout of the database I work on[1], an insane amount of CPU goes to processing of partial index predicates. If the improvements are anywhere near what is being described in the mailing list, we're going to st…

It should be interesting, but LLVM jit brings in a substantial up front cost. Maybe Andres found a way to mitigate it.

The plan is to just use interpretation of expressions for short-running queries. I've spent a lot of work in this release cycle making that faster, and allowing for an easier integration w/ less duplication between JIT and interpreted evaluation. There's two approaches for deciding when to JIT: a) based on query cost estimation. b) just do it after a threshold number of executions. I've not yet decided / experimented enough to make a call.

Additionally, at a later stage, caching of compiled programs will play a role, to reduce the frequency of llvm invocation.

Re: PostgreSQL Parallel Query v2

#24

The relationship between EnterpriseDB and the at-large Postgres community has always intrigued me. Why do they turn over to the community what they could make part of their proprietary fork? Maybe they make a lot of money consulting and not just licenses and general at-large adoption is good for that side of business.

They do, they for example provide fork of Postgres that is compatible with Oracle which is not free.

Re: PostgreSQL Parallel Query v2

#25

Earlier quoted context omitted.

Logical replication doesn't matter as much to us. We are using CitusDB (in other words sharded Postgres), and we have our own code for handling replication.

Citus should already be saturating i/o during a scan unless you're only hitting one shard. What am I missing about your use case?

In many cases all your data is actually in memory, especially when you can distribute your workload across many machines. If all your data is in memory, you'll very often hit CPU bottlenecks before memory bandwidth issues. Partially because bandwidth, disregarding latency, is just quite high, partially because expression evaluation in postgres isn't that well optimized (yet).

Re: PostgreSQL Parallel Query v2

#27

The relationship between EnterpriseDB and the at-large Postgres community has always intrigued me. Why do they turn over to the community what they could make part of their proprietary fork? Maybe they make a lot of money consulting and not just licenses and general at-large adoption is good for that side of business.

Can't be for certain but one possible large factor is it reduces diffs from the upstream if you commit as much as you can back upstream. It's a major engineering drag to deal with that.

Re: PostgreSQL Parallel Query v2

#28

Earlier quoted context omitted.

Logical replication doesn't matter as much to us. We are using CitusDB (in other words sharded Postgres), and we have our own code for handling replication.

Citus should already be saturating i/o during a scan unless you're only hitting one shard. What am I missing about your use case?

We shard our data by customer and individual queries touch only data for one customer. We do split a customer's data into multiple shards, but the number of shards we would need to split each customer into is not worth the overhead of maintaining that many shards.

Re: PostgreSQL Parallel Query v2

#29

Besides parallel queries (specifically parallel index scans), the Postgres 10 feature I'm most looking forward to is the work Andres Freund has done on using LLVM to JIT compile expressions[0]. Based on the the layout of the database I work on[1], an insane amount of CPU goes to processing of partial index predicates. If the improvements are anywhere near what is being described in the mailing list, we're going to st…

Please share profiles on the list. Without known problematic production workloads, it's very hard for us to optimize appropriately. You mention predicate evaluation - it's possible this won't be affected by these changes (the index internals use lower level accesses than generic expression evaluation).

Re prefetching: PG does that for bitmap index scans, if you enable effective_io_concurrency > 1.

Re: PostgreSQL Parallel Query v2

#30

Earlier quoted context omitted.

will oracle be around in 10-20 years? will red hat? which will experience growth?

All things being equal it would take red hat ~30 quarters to make 2bn dollars, or roughly 7 years. 7 years to make what Oracle makes in a quarter. I don't really have a dog in this fight but just FYI I think you missed ops point.

RedHat contributes to the common good, Oracle for a few millionaires. Therefore they create more value.
Post reply on HN