Thanks to the authors for choosing a license that respect users freedom, on top of being an awesome technical project.
Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
51–60 of 167 posts
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#52Earlier quoted context omitted.
They could simply spend a few months and a few million tokens and get their own port, no? I doubt even 30000x faster would prompt a policy change.
No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions. In all fairness, your average app programmer would have the same problems. That's why it takes so long to learn to be a system programmer and why it takes so long to do anything on a systems source base. For reference, systems are OSes, DBs and compilers (although compilers…
> No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions.
What does it being system code have to do with anything?
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#53Cool project but .. reality is that people will generally not choose pgrust over Postgres, even 5-10 years from now. The problem is not that it may be technically superior and faster by then, it's that it's not built by the trusted Postgres team. There's a lot more to trust than development velocity or performance. It's also about the longevity and continuity of a critical piece of technology.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#54Cool project but .. reality is that people will generally not choose pgrust over Postgres, even 5-10 years from now. The problem is not that it may be technically superior and faster by then, it's that it's not built by the trusted Postgres team. There's a lot more to trust than development velocity or performance. It's also about the longevity and continuity of a critical piece of technology.
People will use it if it really delivers gain and can solve problems PG can't.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#55Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…
How do you know if you're making the right optimizations?
I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but that only sometimes makes the production profiler sample move its needle.
e.g. I've recently been experimenting with using a cursed hybrid model in Marginalia's index, where based on a mincore probe, I switch between mmap and io_uring for reading a cluster of pseudoadjacent data. There are real tanglible benefits both in the test machine and in prod with this, but the numbers do not agree at all about how the needle moves :P
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#56Earlier quoted context omitted.
Author here. At least for databases, AGPL (or stricter) has become standard. The issue is it's so easy for megacorps (Amazon, Google, etc) to take a permissively licensed product and monetize it at the expense of the original standard. For instance, Mongo, Cockroach, and Materialize have all gone source available. We picked AGPL because it's the best balance between open source and prevents Amazon from just repackagi…
There is nothing wrong with wanting to be compensated for your work, but for people like myself which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available. I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prev…
what exactly are your concern of using agpl db server? It doesn't force you to make rest of your system agpl.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#57Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…
How do you know if you're making the right optimizations? I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but…
I would probably dig into the reasons for the differences in the benefit on the test machine and in prod
I had an issue like this for optimizing pgrust. I had an optimization that showed no impact on my test machine (c8g.4xl) and showed a 20% improvement when ran on my mac. It turns out the issue was the instruction cache on the c8g.4xl was being saturated on the test machine but not on my laptop, moving the bottleneck to a different place
If you can consistently reproduce the performance difference, you're already half way there
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#58Earlier quoted context omitted.
Author here. At least for databases, AGPL (or stricter) has become standard. The issue is it's so easy for megacorps (Amazon, Google, etc) to take a permissively licensed product and monetize it at the expense of the original standard. For instance, Mongo, Cockroach, and Materialize have all gone source available. We picked AGPL because it's the best balance between open source and prevents Amazon from just repackagi…
There is nothing wrong with wanting to be compensated for your work, but for people like myself which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available. I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prev…
> which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available
That's up to GCP then. I'm sure OP would be willing to license pgrust to them for an appropriate price.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#59Cool project but .. reality is that people will generally not choose pgrust over Postgres, even 5-10 years from now. The problem is not that it may be technically superior and faster by then, it's that it's not built by the trusted Postgres team. There's a lot more to trust than development velocity or performance. It's also about the longevity and continuity of a critical piece of technology.
Their ai agents will if we keep writing about it
Project managers and Human Resources rolling out overengineered projects will outnumber current software engineers 10 to 1
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#60The project has 2 commits.
2.
Commit #1's message is "hey claude, do a breakthrough" from a week ago and is 1.5M lines. Commit #2 is "blog post" from 4 days ago.
My head is spinning. I don't mind AI stuff or AI enabled stuff but there's gotta be some bar for ending up on HN, and also personal accountability: the lack of humility and honesty sets a new low for me. There is no "we" who "released pgrust 0.2". It's one person cosplaying a serious engineering team doing a mountain of work. The bus factor is 1, and its one you can't trust on the basics.
ex. the first 1/3 of the blog post is bloviating about how a rust for loop is faster at summing 500M numbers on the heap than loading the numbers from a table and summing them.
It leaves me in quite some anguish. This site kept me well-informed and growing for 16 years. It is no longer reliable for that as long as things like this can be the #1 post with 60 comments, with the author here, and no one mentioned any of this.