Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

malisper.me

Post reply on HN
Print view View original

151–160 of 167 posts

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#151

Post by saberience »

malisper wrote: post #92
Seattle3503 wrote: post #89

pgrust looks interesting. Could it be used as a library by someone who wants a new DB for each integration test in their Rust test suite?

One of the new features we recently built is "test mode". This brings cloning a template db from 100ms down to If you're interested in trying it out, please reach out to me at malis@pgrust.com

There is no “we” built it.

You’re cosplaying as someone who could build a database.

In reality you yourself could never build Postgres or Redis or any other database.

You’re not skilled enough or knowledgeable enough and you’re not willing to put the time in so you’ve simple vibe copied Postgres.

Why would anyone ever trust you or this project?

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#152

Post by saberience »

esafak wrote: post #4
ognarb wrote: post #3

Earlier quoted context omitted.

2 commits in the repo both generated by claude. This is AI slop, I wonder where you see good momentum?

4K stargazers in a week.

Stars have long ceased to have any meaning since they can be both automated and bought.

Just look at Garry Tans ai slop prompts which got 60k stars in a few weeks.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#153

Post by malisper »

saberience wrote: post #150

I tested pgrust against my own set of benchmarks a week or two ago and it crashed several times during testing. I would not trust this project further than I could throw it.

Can you file an issue? Our big focus over the next few weeks is to eliminate these issues and that's why we're our formal verification and fuzz testing work

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#154

Post by mindwok »

sgt wrote: post #53

Cool project but .. reality is that people will generally not choose pgrust over Postgres, even 5-10 years from now. The problem is not that it may be technically superior and faster by then, it's that it's not built by the trusted Postgres team. There's a lot more to trust than development velocity or performance. It's also about the longevity and continuity of a critical piece of technology.

I don’t think this is true. There’s plenty of signals for trust - if pgrust keep pouring time, money, and effort into this and get people actually using it even in non critical workloads, it’ll snowball from there. Imagine 2 years from now it’s being used in a couple dozen large companies - people wouldn’t think twice about using it.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#155

Post by mindwok »

Blazara wrote: post #134

I don't understand the use case. I've never seen why Postgres is better than anything else if I'm honest. Even if it is much, much faster, database transactions aren't what slow things down if written well in web applications.

It’s not just speed, it’s also the rich feature set. Trigram search, JSONB, transactions, GIN indexes, full text search - not many databases combine them all into a Swiss Army knife like Postgres does.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#156

Post by mhh__ »

hmokiguess wrote: post #126

I don’t understand. It’s not like we don’t already have faster alternatives, don’t we have things like k and kdb that are many orders of magnitude faster even than this? We use Postgres because it’s fantastic at the scale and problems it solve, if you have an extremely critical system where raw speed is at the core of everything and you’re dealing with petascale then maybe you’re bringing a solution you like to a pro…

There should be a halfway point between kdb and postgres...

So more seriously my thinking is that we should, on purely moral grounds strive to have things be fast and lean. How many years of human misery have been lost to waiting around for python programs for example because the team for years wanted to the interpreter to be simple

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#157

Post by malisper »

rastignack wrote: post #149
malisper wrote: post #29

Earlier quoted context omitted.

I'll need to write up how the scheduler works at some point, but it's heavily based on these papers[0][1]. It solves two different problems. First, it lets us throttle resource-intensive queries. Second, it enables work stealing. If you have idle cores on your machine, we'll assign those cores to running queries to help speed them up. That means if you have an over-provisioned machine, we'll make use of the extra cap…

Interesting. It would need some work to handle all the workloads I’ve faced where you have two applications with different priorities (ie oltp or grid workloads and analytics). In those cases you want to make sure that BI users do not interrupt the transaction processing by assigning them a set of cores, a priority, and work mem limits for example. Looks doable without major changes.

We don't expose the priorities right now, but we have the priorities decay over time. That way faster queries get prioritized over long running queries. That should achieve the behavior you're looking for.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#158

Post by malisper »

mdkdog wrote: post #145

wanted to test this but no joy. "ERROR: convert_string_datum (selfuncs.c): pg_strxfrm leg; C-collation lane only" "thread 'pg:backend:1572' (11650) panicked at crates/backend/optimizer/plan/planner/src/selfuncs.rs:966:9:" i would say is not ready yet.

Can you file an issue? We know there are bugs and the work we're doing with formal verification and fuzz testing is to go through all the code and make sure all of it behaves identically to Postgres

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#159

Post by malisper »

saberience wrote: post #151
malisper wrote: post #92

Earlier quoted context omitted.

One of the new features we recently built is "test mode". This brings cloning a template db from 100ms down to If you're interested in trying it out, please reach out to me at malis@pgrust.com

There is no “we” built it. You’re cosplaying as someone who could build a database. In reality you yourself could never build Postgres or Redis or any other database. You’re not skilled enough or knowledgeable enough and you’re not willing to put the time in so you’ve simple vibe copied Postgres. Why would anyone ever trust you or this project?

> You’re cosplaying as someone who could build a database.

> In reality you yourself could never build Postgres or Redis or any other database.

> You’re not skilled enough or knowledgeable enough and you’re not willing to put the time in so you’ve simple vibe copied Postgres.

What makes you think these things? I've been writing about the Postgres internals and giving talks about Postgres for a decade and have managed a Postgres cluster as big as 1PB of data

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#160

Post by malisper »

NautilusWave wrote: post #144

Reading about the batch optimization, I suspect there'll be trouble ahead for implementing window functions, unless those details were dropped for simplicity.

How so? Window functions work exactly the same way. Postgres processes them one row at a time, and you can batch them the same way as you would with sum.

Window functions do make parallel queries more difficult, but that's a different story.

Post reply on HN