Live data from Hacker News

Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

malisper.me

31–40 of 167 posts

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#31

Earlier quoted context omitted.

If it is indeed 300* faster, I'm sure more rational corporations will rethink their license policy or be left in the dust.

They could simply spend a few months and a few million tokens and get their own port, no? I doubt even 30000x faster would prompt a policy change.

Are you suggesting the AI just rewrites the whole thing under a different license? There’s no way that’s not more dicey than the AGPL license.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#32

Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…

what is your vision of this project? Do you think pgrust will eventually be prod ready?

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#33

Earlier quoted context omitted.

Oh bummer. I was really excited about pgrust but AGPL is a dealbreaker. Not for me personally, but it will never see wide adoption because it’s a banned license in most corporate environments. Lack of path to wide adoption means it’s dead in the water. It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such…

If it is indeed 300* faster, I'm sure more rational corporations will rethink their license policy or be left in the dust.

License policies are made by lawyers not by programmers. And their competition will be in the exact same boat (different lawyers though). AGPL is so toxic that it tends to be checked for during M&A processes so even if the current batch of lawyers is ok with it there is a chance that a later batch of lawyers is not. Given that the target audience for this project is the larger companies you are going to end up with a very nice project and zero actual users or you will end up with AWS et all stealing your work. Databases are very hard to do successfully commercially, at a minimum you should dual license them (AGPL for 'home' use and commercial licensing for parties that will want to buy the upside but they'll demand support and other stuff besides).

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#34

Earlier quoted context omitted.

They could simply spend a few months and a few million tokens and get their own port, no? I doubt even 30000x faster would prompt a policy change.

Are you suggesting the AI just rewrites the whole thing under a different license? There’s no way that’s not more dicey than the AGPL license.

That's exactly what they did here, I don't see the difference.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#35

Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…

what is your vision of this project? Do you think pgrust will eventually be prod ready?

I want to build the best database possible. While Postgres is great, there are a lot of core issues that have been around for over a decade. We're working hard to get pgrust production-ready, and it will definitely be production-ready in the near future. I wouldn't be putting hundreds of thousands of dollars into this project if I didn't think we could build a production-ready database.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#36

Earlier quoted context omitted.

If it is indeed 300* faster, I'm sure more rational corporations will rethink their license policy or be left in the dust.

They could simply spend a few months and a few million tokens and get their own port, no? I doubt even 30000x faster would prompt a policy change.

No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions. In all fairness, your average app programmer would have the same problems. That's why it takes so long to learn to be a system programmer and why it takes so long to do anything on a systems source base. For reference, systems are OSes, DBs and compilers (although compilers are very different in many ways).

Also, you can successfully sell a systems project that is only 10% faster. 30000x faster and they are throwing illegal and debauched things through your window to get access to your improvements.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#37

Earlier quoted context omitted.

Oh bummer. I was really excited about pgrust but AGPL is a dealbreaker. Not for me personally, but it will never see wide adoption because it’s a banned license in most corporate environments. Lack of path to wide adoption means it’s dead in the water. It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such…

If it is indeed 300* faster, I'm sure more rational corporations will rethink their license policy or be left in the dust.

Its 300x faster for certain tests. I could pretty easily craft tests that do this on two different systems. The author mentioned this when they talked about being able to fit an entire ResultSet into memory. That's the real trick with performance. Very few workloads are CPU bound anymore (linear algebra on the CPU for example). Almost all workloads are memory bound. So its all about moving data from memory to network, back to memory and back to network, over and over again through your microservices or DBs. If the entire working set can fit in memory, you get at least a 10x performance boost. If you have to keep even a part of the working set on disk, its a huge performance loss. And the larger fraction of the working set on disk, the worse the performance loss.

PS Learn how DBs do joins for more information. Specifically the differences between hash joins, merge joins and nested loop joins. They are basically fancy ways to page part of your working set to disk at huge performance penalties.

PPS As memory gets more expensive, these techniques get more valuable. When it gets cheap, they lose value.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#38

Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…

This is a great project. Thank you!

A question on 20s postgresql time - It does not look like you are accounting for reading data from disk? Wouldn't the aggregation query have to load data from disk first? Or is it somewhat guaranteed that the table is already in memory? The Rust version is clearly in memory (I am no rust expert, so that may not even be actually in memory, if its a generator).

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#39

Earlier quoted context omitted.

If it is indeed 300* faster, I'm sure more rational corporations will rethink their license policy or be left in the dust.

License policies are made by lawyers not by programmers. And their competition will be in the exact same boat (different lawyers though). AGPL is so toxic that it tends to be checked for during M&A processes so even if the current batch of lawyers is ok with it there is a chance that a later batch of lawyers is not. Given that the target audience for this project is the larger companies you are going to end up with a…

This is the case when you have your own datacenter. This isn't as big a problem in the cloud. There are ways to write licenses that prevent cloud providers from stealing while allowing customers to use the software and being required to pay for it. The problem with the AGPL has to with its viral nature, not its provisions to prevent cloud vendor theft.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#40

pgrust seems to have good momentum. AGPL is an odd license for a non web project. Postgres is MIT-like, and that drove it's adoption. Have pgrust folks reconsidered this? Else, IMO we can have an independant rust port of pgrust, which can be MIT, which will garner more attention.

Oh bummer. I was really excited about pgrust but AGPL is a dealbreaker. Not for me personally, but it will never see wide adoption because it’s a banned license in most corporate environments. Lack of path to wide adoption means it’s dead in the water. It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such…

> It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it?

Wait, sorry, you're asking why make something enterprise customers might pay for, and then not give it away to them for free?

Post reply on HN