Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

malisper.me

Post reply on HN
Print view View original

61–70 of 169 posts

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#61

Post by wffurr »

jnwatson wrote: post #15
malisper wrote: post #11

Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…

The floating point comparison bug is nightmare fuel. I could look at that for years and never spot the mistake.

Fuzzers are brilliant at this and produce all kinds of insane floating point inputs.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#62

Post by hobofan »

malisper wrote: post #10
cognitiveinline wrote: post #2

pgrust seems to have good momentum. AGPL is an odd license for a non web project. Postgres is MIT-like, and that drove it's adoption. Have pgrust folks reconsidered this? Else, IMO we can have an independant rust port of pgrust, which can be MIT, which will garner more attention.

Author here. At least for databases, AGPL (or stricter) has become standard. The issue is it's so easy for megacorps (Amazon, Google, etc) to take a permissively licensed product and monetize it at the expense of the original standard. For instance, Mongo, Cockroach, and Materialize have all gone source available. We picked AGPL because it's the best balance between open source and prevents Amazon from just repackagi…

> If AGPL is an issue for anyone, we would be happy to dual-license under a commercial license.

You should be setting up a CLA process then ASAP if you are looking to take outside contributions.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#63

Post by marginalia_nu »

malisper wrote: post #57
marginalia_nu wrote: post #55

Earlier quoted context omitted.

How do you know if you're making the right optimizations? I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but…

I would probably dig into the reasons for the differences in the benefit on the test machine and in prod I had an issue like this for optimizing pgrust. I had an optimization that showed no impact on my test machine (c8g.4xl) and showed a 20% improvement when ran on my mac. It turns out the issue was the instruction cache on the c8g.4xl was being saturated on the test machine but not on my laptop, moving the bottlene…

I'm pretty sure the reason for the difference is that production machine exists in a state of mixed memory residency and low grade resource contention that is incredibly hard to replicate in a test scenario (as the moment you start making queries the pages warm up, and the test becomes unreliable).

The hard part about optimizing this type of code, IMO, is that there are so many cache layers, both in the CPU and the OS and sometimes in the storage medium. You can warm all of those caches up, but then you're testing a nonsense scenario that will basically never happen in a realistic scenario, where wall clock time is what matters, and not much the CPU is working or how many IOPS you're pushing.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#64

Post by wffurr »

refulgentis wrote: post #60

The project has 2 commits. 2. Commit #1's message is "hey claude, do a breakthrough" from a week ago and is 1.5M lines. Commit #2 is "blog post" from 4 days ago. My head is spinning. I don't mind AI stuff or AI enabled stuff but there's gotta be some bar for ending up on HN, and also personal accountability: the lack of humility and honesty sets a new low for me. There is no "we" who "released pgrust 0.2". It's one p…

The second line of commit #2's message is "You can find the actual git history at the v0.2 github tag." which in turn has almost 6000 commits.

I think it's a weird way to handle git history versus squashing feature branches into single commits, but it's not just one Claude session slapped up on GitHub. The OP also has a post above about their exhaustive testing which has uncovered a goodly number of bugs in Postgres itself, too. I think it's fair to say they're putting in a good amount of work on this.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#65

Post by wrs »

cognitiveinline wrote: post #2

pgrust seems to have good momentum. AGPL is an odd license for a non web project. Postgres is MIT-like, and that drove it's adoption. Have pgrust folks reconsidered this? Else, IMO we can have an independant rust port of pgrust, which can be MIT, which will garner more attention.

Good news, we now know up front what an independent port would cost, and it’s not much. So no reason AWS, Google, and friends couldn’t bang out their own port if they want, binary-compatible with this one.

What we don’t know for sure is whether there is any copyright for LLM-generated code. The license might be irrelevant!

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#68

Post by claytonjy »

wiradikusuma wrote: post #67

I think in addition to making it faster, it would be useful if it could be made "leaner," e.g. can run better on lower-spec hardware than PG.

does one not imply the other? if it can run faster in the same hardware, it should also run as fast on lower spec hardware

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#69

Post by sgt »

yieldcrv wrote: post #59
sgt wrote: post #53

Cool project but .. reality is that people will generally not choose pgrust over Postgres, even 5-10 years from now. The problem is not that it may be technically superior and faster by then, it's that it's not built by the trusted Postgres team. There's a lot more to trust than development velocity or performance. It's also about the longevity and continuity of a critical piece of technology.

Their ai agents will if we keep writing about it Project managers and Human Resources rolling out overengineered projects will outnumber current software engineers 10 to 1

That's a pretty gloomy view

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#70

Post by refulgentis »

wffurr wrote: post #64
refulgentis wrote: post #60

The project has 2 commits. 2. Commit #1's message is "hey claude, do a breakthrough" from a week ago and is 1.5M lines. Commit #2 is "blog post" from 4 days ago. My head is spinning. I don't mind AI stuff or AI enabled stuff but there's gotta be some bar for ending up on HN, and also personal accountability: the lack of humility and honesty sets a new low for me. There is no "we" who "released pgrust 0.2". It's one p…

The second line of commit #2's message is "You can find the actual git history at the v0.2 github tag." which in turn has almost 6000 commits. I think it's a weird way to handle git history versus squashing feature branches into single commits, but it's not just one Claude session slapped up on GitHub. The OP also has a post above about their exhaustive testing which has uncovered a goodly number of bugs in Postgres…

I cloned it. The 6000 commits are the problem, not the defense.

5,940 commits, one author, 29 days. 5,067 of them, 85%, have a `Co-Authored-By: Claude` trailer. Busiest day is 1,393 commits, running 60-105/hour for ~20 hours straight. Commit messages reference .claude/skills/fleet/ and agent worktree "lanes". That's an unattended agent fleet committing once a minute around the clock. Commit count used to mean review time. Here it means GPU time.

Also: the v0.2 tag shares no common ancestor with main. git merge-base fails. "The actual git history" is an orphan branch grafted in after the fact.

And the testing rigor we're crediting them for should be weighed against the headline claim: 300x faster than Postgres, ahead of ClickHouse: fastest analytical DB on earth, one guy, one month.

Most damning: the Postgres baseline ran with *max_parallel_workers_per_gather = 0*. Parallel engine vs. deliberately-hobbled single-core Postgres, headline says 300x.

Using AI is fine. But "look how many commits" is precisely the signal this workflow is built to fake.

Post reply on HN