Can some of these optimization get back propagated to Postgres?
Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
111–120 of 167 posts
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#112Earlier quoted context omitted.
I want to build the best database possible. While Postgres is great, there are a lot of core issues that have been around for over a decade. We're working hard to get pgrust production-ready, and it will definitely be production-ready in the near future. I wouldn't be putting hundreds of thousands of dollars into this project if I didn't think we could build a production-ready database.
If you are at that, add a native TTL feature as well. I think it’s been requested for decades.
What do you mean by native TTL? Would that be when rows are automatically deleted if they aren't touched after a certain period of time?
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#113Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…
I really hope your project succeeds so that you can sit, sip delicious coffee with your legs on a high desk, and laugh really loud, like a gorilla, at all the haters from this and previous HN threads.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#114Cool project but .. reality is that people will generally not choose pgrust over Postgres, even 5-10 years from now. The problem is not that it may be technically superior and faster by then, it's that it's not built by the trusted Postgres team. There's a lot more to trust than development velocity or performance. It's also about the longevity and continuity of a critical piece of technology.
We still have companies that use COBOL and mainframes due to their longevity and continuity. There is no reason Postgres and pgrust cannot both flourish. I can pick and choose which one to use according to its use case.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#115Question, does having it in pure rust, opens possibility of embedding pgrust directly into binary, making it an alternative to SQLite/turso?
I'd help with testing if you helped them add this. I'd be interested in this.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#116300x if it is true you will be just busy dealing with you customers rather than pitching here. Also since it is a vibe coded project, if you are really that good , you should even need to related yourself with Postgres. Who will want related itself to something that is 300x slower than itself?
You use stability of PostgreSQL to open a pitch. Also most people use PostgreSQL because someone out there was a fan, proponent and champion of Postgres. Otherwise they'd be on MySQL or Oracle.
And your sales folks would call and say: "No need to change anything, we still run PostgreSQL, and ours is just called pgrust, but it's N times as fast".
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#117I'm really happy seeing this project. Not sure if this helps you gain $$$ customers, but stupid thing that turns out very difficult in PG is making this fast:
SELECT COUNT(*) FROM large_text_db WHERE X
Where X is something that must be matched exactly. X can be FTS query on FTS-indexed table, but the way COUNT() works in PG is that it's impossible to make it fast. Over large tables, lets say 1B+ rows, it can be very very slow.
Example use case is: searching through a hospital DB of reports that have "pancreatic cancer" in them. This is trivial in SQLite, but in PG it's hard.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#118Earlier quoted context omitted.
> No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions. What does it being system code have to do with anything?
The brittleness and difficult of writing the code. The extra complexity. The cost of bugs. And ease of creating performance degradations. If you are even asking that question, you probably don't have the experience to even have an opinion on this. Most systems code has to be "canonical". That means it can't be improved. Imagine the most difficult PR review you have ever had. Now give the reviewers OCD and an extra 30…
> The brittleness and difficult of writing the code. The extra complexity. The cost of bugs. And ease of creating performance degradations. If you are even asking that question, you probably don't have the experience to even have an opinion on this.
You didn't express anything unique to systems engineering.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#119Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…
How do you know if you're making the right optimizations? I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but…
Optimizing a multi-layer cache system is a Complex System task with no subset you can reduce to a simpler version of the problem. PG has a 3 layer cache--database, OS, storage--and when you combine that with concurrency and locality, you have a giant 5 dimensional space to map out. Building the right set of benchmarks for PG with guidance for the most important parts to navigate fro that space has kept me busy for 20 years now.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#120Earlier quoted context omitted.
That's exactly what they did here, I don't see the difference.
This is the difference. This guy took an existing source base, had Claude find specific bugs, then had Claude fix a specific bug which was then reviewed by a person. We also don't know if these changes introduce new problems yet. You are suggesting letting Claude write an entirely new source base. That's light-years away from what happened here.
> light-years away from what happened here
There is also zero evidence a human has reviewed a meaningful amount of the code generated for the port, let alone the original C code from Postgres, given the size/scale of the project and the how quickly it was produced.