Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

github.com

Post reply on HN
Print view View original

171–180 of 756 posts

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#171

Post by bozdemir »

OtomotO wrote: post #48
bozdemir wrote: post #35

Earlier quoted context omitted.

I'd %100 prefer an opus 4.8 rewrite over %99 of the time. Unless Fabrice Bellard is rewriting the stuff I need, I'd prefer AI over a human coder.

AI is an average coder. It was trained on all code the code that could be found. Not just code written by genius programmers like Carmack and Bellard. Given that it's average, I'd prefer a human coder above average :)

I dont think Opus 4.8 is an average coder, with my own experience (I have coded 20 + years before even llms existed) it is anything but average. I don't think training data alone determines the success of these models, there are lots of reinforncement learning principles and fine tuning takes place, a crappy code in the dataset doesnt hold those llms scoring high in benchmarks, I dont think an average programmer can score 70% (opus 4.8) in SWE Bench Pro, which is a good one.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#172

Post by tudorg »

malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

> - Is ~300x faster than Postgres on analytical workloads. Right now it's 2x slower than Clickhouse on clickbench and I think it's possible to get faster than Clickhouse

That sounds like you are storing the data in a columnar format? Or do you do both row and columnar?

In a somewhat similar (yet also quite different) effort, I've been working on δx, a Postgres extension that compresses the data in a columnar format stored in normal Postgres tables (so replication, crash recovery, pg_dump, etc. still work normally). https://github.com/xataio/deltax

It is currently about 30-40% slower than ClickHouse (single node, ofc). The PR to add it to clickbench was just accepted, so you can see the comparison here: https://benchmark.clickhouse.com/#system=+liH|_etx|gQ|saB&ty...

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#173

Post by BadBadJellyBean »

arka2147483647 wrote: post #79
ottavio wrote: post #33

Why should a developer use this for anything beyond a pet project? Just because it is written in Rust? All these "rewritten in rust" projects only reinforce the idea that a significant part of the rust community consists of software talibans and not of engineers who must deliver something that works and is reliable over time.

Often the biggest blocker on moving to a new programming language, is the cost of re-writing everything. Cue some story here on a bank or airline somewhere still relying on cobol backend servers. These LLM conversions really seem to make modernization of large parts software layers possible!

It's not enough to do a rewrite. Someone has to maintain it. Such a huge codebase with literally zero experts is unmaintainable. There is no one who knows how the internals work.

Sure you could keep vibe coding it but I wouldn't bet my data on that. A database needs to be rock solid.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#174

Post by malisper »

jl6 wrote: post #167
malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

"Is 50% faster than Postgres on transaction workloads" - That is a very big claim! 50% faster on everything? Is it a strict improvement across the board or are there tradeoffs that make some workloads slower?

The 50% is specifically on percona-tpcc[0]. I got there through a mix of batching (postgres processes a row at a time), prefetching, and several handful of other optimizations.

  [0] https://github.com/Percona-Lab/sysbench-tpcc

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#175

Post by mebcitto »

malisper wrote: post #168
mebcitto wrote: post #13

Does it support the extension ecosystem? Or would extensions need to be rewritten as well?

It is theoretically possible to have a Rust port of Postgres support extensions. If you make all the relevant functions and structures ABI compatible with Postgres, extensions should work. The issue is the moment you're dealing with C pointers and C strings, pretty much all the code you have to write is unsafe.

Perhaps pure pgrx extensions would make sense as a first target?

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#176

Post by gnull »

malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

What was your methodology and structure in making the prompts for the rewrite? Did you let the LLM roam in all of the codebase and tests from the beginning, or revealed things to it gradually in some way?

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#177

Post by 72deluxe »

BadBadJellyBean wrote: post #173
arka2147483647 wrote: post #79

Earlier quoted context omitted.

Often the biggest blocker on moving to a new programming language, is the cost of re-writing everything. Cue some story here on a bank or airline somewhere still relying on cobol backend servers. These LLM conversions really seem to make modernization of large parts software layers possible!

It's not enough to do a rewrite. Someone has to maintain it. Such a huge codebase with literally zero experts is unmaintainable. There is no one who knows how the internals work. Sure you could keep vibe coding it but I wouldn't bet my data on that. A database needs to be rock solid.

This seems to be the issue with using LLMs for any code generation. Even with my own code bases that I've written entirely by hand over years, if I use AI to implement anything, I don't go through the mental model of architecting it, so I don't know how it works. I can only imagine this to be far, far worse for large code bases maintained by a team of people who are all using AI.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#178

Post by marcus_holmes »

Ameo wrote: post #20
tormeh wrote: post #5

Woah! AGPL? That's interesting. I think Postgres has shown an open source SQL server didn't need a copy-left license to develop sustainably, so I'm not entirely aure about that, but I do like the license in general.

When the software consists entirely of ~$1000 worth of Claude credits and ~40 hours of developer time prompting and curating it, literally what does it matter what license the resulting 100k LoC artifact is provided under? Copyleft and the whole software licensing ecosystem only matter when producing that software actually requires serious human effort and dedication.

> Copyleft and the whole software licensing ecosystem are only applicable when producing software that actually required human effort.

Fixed that for you. Code generated by an LLM is not copyrightable (because copyright only protects human effort), so the codebase is automatically public domain and cannot be licensed at all.

They could theoretically copyright the prompts that they used, but as that's not part of the output, and the output doesn't deterministically arise from those prompts, they'd struggle to use that to back a copyright claim.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#179

Post by ykonstant »

tgv wrote: post #136
evil-olive wrote: post #107

> The goal is to make Postgres easier to change from the inside uh-huh, sure. you want to show off "look what the LLM can do / look what I burned a bunch of tokens on"? you want to brag about how your LLM-generated slop is somehow more maintainable than the original because blah blah blah Rust? here [0] is the version history of Postgres. pick a version from the past. let's say 14.x because it's the most current that…

It's not just unsafe, it's this: let r = unsafe { &*p }; It looks as if it's building structs out of information in (mutable pointers) to other structs without an Rc in sight. Which makes sense for a C parser: you've got a table with data, so you just link to it. It's fast, and when you know you're not going to touch it, it's safe. But this doesn't make the Rust code any better than the C code.

Genius! Rewrite Postgres in Rust by rewriting Rust in C!

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#180

Post by [deleted] »

scotty79 wrote: post #105

Rewrites in Rust are kinda impressive. This language with its move semantics and close ownership tracking is very different from every other language. To create a rewrite in it, you have to rearchitect the code. There is not as much freedom there when it comes to where to keep what and where you can pass what as it is in other languages.

[deleted]
Post reply on HN