Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

github.com

Post reply on HN
Print view View original

611–620 of 756 posts

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#611

Post by tudorg »

FreakLegion wrote: post #414
malisper wrote: post #204

Earlier quoted context omitted.

Yep! The new version of pgrust supports batch based execution and a columnar format. I'm curious how you got δx to perform that well? From what I've seen a columnar layout only gets you part of the way and really good parallelism and really fast hash tables seem to make up a significant portion of why Clickhouse is faster.

pg_mooncake (now effectively abandoned due to being acquired by Databricks, but still up at https://github.com/Mooncake-Labs/pg_mooncake ) pulled the DuckDB engine into Postgres wholesale, if I remember right. pg_lake also uses DuckDB but keeps it external, routing through Postgres and managing Iceberg tables (but not the data itself) there ( https://github.com/Snowflake-Labs/pg_lake ). Both of these were neck and ne…

Actually δx is faster than the "duckdb embedded in postgres" options: https://benchmark.clickhouse.com/#system=+_etx|_b|_i)|dula|pnc&type=-&machine=-6t|ca2|6ax|g4e|6ale|3al&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-

Plus all the normal Postgres features work as expected: physical/logical replication, crash recovery, pg_dump/pg_restore, etc.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#613

Post by lucideer »

Jean-Papoulos wrote: post #504
sdevonoes wrote: post #251

Don’t understand these rewrites. - typically they are behind a single person. That’s usually bad because of spf - typically they are achieved in a very short amount of time, so the author hasn’t acquired any discipline in creating the project. That means it’s unlikely the author is going to stick to the project in the mid and long term - anyone that wants to contribute to the project needs to pay. Needs to pay tokens…

It's useful to show the actual team that it's possible. From there, they can make the decision of whether to go the bun route with more information.

I'm not sure I see the value in "showing the team it's possible". I would presume the team are intelligent enough to be well aware that it's possible, given tradeoffs. And as it's clear those tradeoffs have been "traded" in this case, it doesn't seem like having the knowledge confirmed is particularly surprising.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#614

Post by malisper »

kaoD wrote: post #578
diziet wrote: post #576

Earlier quoted context omitted.

I am fully aware what Cargo.lock is. What I am surprised at is how many dependencies there are.

Sorry, the wording confused me. The project itself seems to be structued around 1.4k micro-crates[0] which I admit is a bit weird. Rust's compilation unit is the crate unlike C's per-file compilation unit, so if this was a 1:1 AI-assisted translation from the original Postgres source this might be an artifact of the translation. [0] https://github.com/malisper/pgrust/blob/main/Cargo.toml

This is correct. We first used c2rust to translate the C into unsafe Rust. The generated Rust code had one crate per C compilation unit. We then took the unsafe unidiomatic Rust and one crate at a time converted it to safe Rust.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#615

Post by fg137 »

davedx wrote: post #584
sdevonoes wrote: post #251

Don’t understand these rewrites. - typically they are behind a single person. That’s usually bad because of spf - typically they are achieved in a very short amount of time, so the author hasn’t acquired any discipline in creating the project. That means it’s unlikely the author is going to stick to the project in the mid and long term - anyone that wants to contribute to the project needs to pay. Needs to pay tokens…

This is very critical of an open source project that the maintainer didn't even post here? "Status: pgrust is not production-ready yet. It is not performance optimized yet." The maintainer is not suggesting you use this for anything yourself. So why do you care about spf or (lol) his "discipline in creating the project"?

Because chances are that it's never going to be production ready, never trustworthy enough to be used by anything serious, and the project will eventually be archived and become at best a (extremely inefficient) learning experience for the author, or at worst a total waste of tokens. I could be completely wrong about this specific project, but most of these projects are like that, and statistics don't lie.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#616

Post by gen2brain »

davedx wrote: post #584
sdevonoes wrote: post #251

Don’t understand these rewrites. - typically they are behind a single person. That’s usually bad because of spf - typically they are achieved in a very short amount of time, so the author hasn’t acquired any discipline in creating the project. That means it’s unlikely the author is going to stick to the project in the mid and long term - anyone that wants to contribute to the project needs to pay. Needs to pay tokens…

This is very critical of an open source project that the maintainer didn't even post here? "Status: pgrust is not production-ready yet. It is not performance optimized yet." The maintainer is not suggesting you use this for anything yourself. So why do you care about spf or (lol) his "discipline in creating the project"?

But this is also just silly. The maintainer here did nothing (just a little chat here and there), but is sure that only optimizations are missing for this to be production-ready. How so, based on what exactly?

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#617

Post by menaerus »

win311fwg wrote: post #388
film42 wrote: post #262

Earlier quoted context omitted.

A thread per connection is a almost always the correct decision for performance, but by choosing a process per connection, postgres is able to let you load whatever sketchy extensions you want. Worst case you crash the process, not the database. It would be nice if you could strike a balance so a segfaul in the extension only crashes a small percentage of connections, not the whole thing.

An OS thread per connection can be fine for performance if you don't have to scale your connections, but if you don't need to scale connections why have connections at all? Databases are even more performant when you eliminate connection overhead entirely.

Thread pools

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#618

Post by fg137 »

anonzzzies wrote: post #466
sdevonoes wrote: post #251

Don’t understand these rewrites. - typically they are behind a single person. That’s usually bad because of spf - typically they are achieved in a very short amount of time, so the author hasn’t acquired any discipline in creating the project. That means it’s unlikely the author is going to stick to the project in the mid and long term - anyone that wants to contribute to the project needs to pay. Needs to pay tokens…

It's not just a rewrite ; it has improvements. I did the same thing for fun for the same reason; I wanted to see if I can improvements on some of the legacy design stuff and, especially, the stuff PG people have told us that it cannot be done differently. It can. I would not put it in production, but it thought me a lot about the internals of databases. To keep my brain happy in the age of LLMs, I implement database…

> I would not put it in production

Nobody else would either.

If this is meant for personal learning, do it that way and make it clear that others should not even consider using this project.

In fact, even for personal learning it's wasteful. You can learn so much about database without a rewrite like this.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#619

Post by fg137 »

literalAardvark wrote: post #605
sgt wrote: post #599

Earlier quoted context omitted.

An academic exercise for sure. The Postgres team won't use this and take it forward, hence it will go stale and rot within months.

Or someone who likes it could run codex-get -y upgrade on it once every two weeks and it'll be fine for as long as you can afford the tokens.

Yet nobody is going to use it in production.

So what's the point?

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#620

Post by abbefaria27 »

Setting aside the “why” that everyone is so focused on, I want to know, how? I can’t get Claude to do anything even a fraction of this complexity. How are people setting up their agents, Claude.md, etc to do such big projects? There are a lot of lessons to be learned.
Post reply on HN