PostgreSQL Parallel Query v2 • Hacker News Board

Re: PostgreSQL Parallel Query v2

#41

Post by jeltz » Wed, Mar 15, 2017, 11:28 PM UTC

koolba wrote: post #6

pstuart wrote: post #5

Earlier quoted context omitted.

I think their secret sauce is Oracle compatibility, so if Postgres runs better, they run better.

And the Oracle compatibility piece breaks so much that it would never get merged into community Postgres.

There are also plenty of devs in the community, outside the EnterpriseDB employees, who do do not want to pollute core PostgreSQL with alternative ways to do things it already can do just because of Oracle compatibility. Aliases have been added for ISO SQL compatibility, but adding features purely (i.e. features with zero value unless you are migrating from Oracle) for Oracle compatibility is not something core PostgreSQL does.

Re: PostgreSQL Parallel Query v2

#42

Post by malisper » Thu, Mar 16, 2017, 12:55 AM UTC

manigandham wrote: post #34

malisper wrote: post #28

Earlier quoted context omitted.

We shard our data by customer and individual queries touch only data for one customer. We do split a customer's data into multiple shards, but the number of shards we would need to split each customer into is not worth the overhead of maintaining that many shards.

Why? Is this a security issue? If you're sharing the same distributed table then sharding by customer is limiting the number of nodes/scans in parallel that can run.

Having millions of shards is a bit of a pain ops wise. There is also some other costs to having lots of shards. For example, the current Citus distribution planner takes time linear in the number of total shards. This isn't an issue 99% of the time, but since we use a composite type as our distribution key, we have an extremely high constant overhead for planning time. I believe the current median Citus Master planning time is ~350ms. We've talked to Citus about this and have discussed possible solutions. Currently we are trying to purge the composite type from our database.

Re: PostgreSQL Parallel Query v2

#43

Post by electrum » Thu, Mar 16, 2017, 1:37 AM UTC

jordanthoms wrote: post #20

fnord123 wrote: post #16

Earlier quoted context omitted.

It should be interesting, but LLVM jit brings in a substantial up front cost. Maybe Andres found a way to mitigate it.

Is it significant when you are talking about analytical queries which are running for minutes or hours? Seems like it wouldn't be a huge amount of code to JIT compared to the payoff?

Presto, which is designed for analytics, does JIT to bytecode on every query and still works fine on sub-second queries. One potential difference with LLVM is that the JVM will profile and only generate machine code for hot code, so short queries might use JVM interpreted bytecode.

But you are absolutely correct that for long running analytic queries, the up front cost of generating optimized code pays off.

Re: PostgreSQL Parallel Query v2

#44

Post by fleetside72 » Mon, Mar 20, 2017, 4:15 PM UTC

rodgerd wrote: post #11

fleetside72 wrote: post #2

The relationship between EnterpriseDB and the at-large Postgres community has always intrigued me. Why do they turn over to the community what they could make part of their proprietary fork? Maybe they make a lot of money consulting and not just licenses and general at-large adoption is good for that side of business.

Well, Red Hat have demonstrated you can make a lot of money by open sourcing things (including buying proprietary software like ManageIQ and oVirt and open sourcing it). Given that EDB has ex-RHAT folks involved in setting it up, I imagine they're trying to replay the model.

I don't see how Red Hat has open sourced anything, the direction they are taking things is proprietary, not open.