Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

malisper.me

Post reply on HN
Print view View original

21–30 of 166 posts

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#21

Post by xyzzy_plugh »

jnwatson wrote: post #16
cognitiveinline wrote: post #13

Earlier quoted context omitted.

Sure, that's your prerogative, and kudos for not talking up open source. I'm not amazon size so can't use it, and AGPL is a no go for DB, don't want to be forced to open source my app because I use this! Will await a MIT based fork myself.

Why would AGPL force you to open source your app? Unless you literally compile your app with pgrust by modifying the pgrust source code, you're safe. Clients aren't bound by the AGPL because they aren't derived works.

Here we go again.

AGPL is untested in courts. There is no definitive definition of what could be considered within the blast radius such that it would require AGPL licensing.

There's a reason AGPL is banned at Google and most sane companies. It's simply too dangerous.

You can't simply say "clients aren't bound" because it depends.

I'd rather see the BSL used here to be perfectly honest. At least it's simple.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#22

Post by lizimo »

malisper wrote: post #11

Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…

Is `pgrcolumnar` the default storage layout for tables? It would be cool if the same storage engine outperforms vanilla Postgres under both OLTP and OLAP workloads.

AlloyDB from Google Cloud uses columnar storage like a secondary index, while the relations are still stored in TOAST.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#23

Post by xyzzy_plugh »

guenthert wrote: post #20
appplication wrote: post #7

Earlier quoted context omitted.

Oh bummer. I was really excited about pgrust but AGPL is a dealbreaker. Not for me personally, but it will never see wide adoption because it’s a banned license in most corporate environments. Lack of path to wide adoption means it’s dead in the water. It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such…

If it is indeed 300* faster, I'm sure more rational corporations will rethink their license policy or be left in the dust.

They could simply spend a few months and a few million tokens and get their own port, no?

I doubt even 30000x faster would prompt a policy change.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#24

Post by malisper »

lizimo wrote: post #22
malisper wrote: post #11

Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…

Is `pgrcolumnar` the default storage layout for tables? It would be cool if the same storage engine outperforms vanilla Postgres under both OLTP and OLAP workloads. AlloyDB from Google Cloud uses columnar storage like a secondary index, while the relations are still stored in TOAST.

pgrcolumnar is not the default storage method. Right now, it's exposed as a table access method. There's lots of design space for how to do this so I want to avoid pre-committing to anything

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#26

Post by rastignack »

I would be interested about a more detailed architecture overview of the io scheduler (like this: https://www.scylladb.com/2021/04/06/scyllas-new-io-scheduler...) and the thread scheduler.

PostgreSQL has historically been bad at managing the noisy neighbor problem, but with thread pools, and io priorities, it can be solved.

Has this been tackled here ?

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#28

Post by bee_rider »

jnwatson wrote: post #15
malisper wrote: post #11

Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…

The floating point comparison bug is nightmare fuel. I could look at that for years and never spot the mistake.

On the bright side it could probably run for years without hitting the mistake as well. But it is nice to get it out of there.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#29

Post by malisper »

rastignack wrote: post #26

I would be interested about a more detailed architecture overview of the io scheduler (like this: https://www.scylladb.com/2021/04/06/scyllas-new-io-scheduler... ) and the thread scheduler. PostgreSQL has historically been bad at managing the noisy neighbor problem, but with thread pools, and io priorities, it can be solved. Has this been tackled here ?

I'll need to write up how the scheduler works at some point, but it's heavily based on these papers[0][1]. It solves two different problems. First, it lets us throttle resource-intensive queries. Second, it enables work stealing. If you have idle cores on your machine, we'll assign those cores to running queries to help speed them up. That means if you have an over-provisioned machine, we'll make use of the extra capacity to speed your queries up.

[0] https://15721.courses.cs.cmu.edu/spring2016/papers/p743-leis...

[1] https://db.in.tum.de/~kohn/papers/query-scheduling-sigmod21....

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#30

Post by f311a »

Whitespace wrote: post #5
ognarb wrote: post #3

Earlier quoted context omitted.

2 commits in the repo both generated by claude. This is AI slop, I wonder where you see good momentum?

main indeed has two commits, but it clearly states the location of the rest of the commits, so I wouldn't be critical of main itself. hey claude, do a breakthrough You can find the actual git history at the v0.2 github tag. Co-Authored-By: Fable Now we see https://github.com/malisper/pgrust/tree/v0.2 has almost 6000 commits in it, with the very first one on 2026-07-02. That's a lot of token momentum! It's easy to cla…

What's the reason for it? Does not make a lot of sense to keep all the commits elsewhere
Post reply on HN