Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

github.com

Post reply on HN
Print view View original

371–380 of 756 posts

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#371

Post by mattgreenrocks »

malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

I am super curious how you went about the port using LLMs. At $WORK we are looking to port code, preferably with LLMs, and it seems daunting, even with a test suite. Do you have an approach that works well for you?

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#372

Post by mulmen »

brianaker wrote: post #335
fpgaminer wrote: post #292

Earlier quoted context omitted.

If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.

A mixture of threads and processes that can be used to match processors, disk I/O, and network interfaces. A very long time ago, there was once a feature called "Data Blades" which tanked a commercial database vendor. A badly behaving blade could bring down the entire database. Most anyone who has been working on databases for a few decades remembers this and makes a point of either not introducing these sorts of fea…

> A very long time ago, there was once a feature called "Data Blades" which tanked a commercial database vendor.

I have no idea what this is and a web search turned up Harbor Freight woodworking tools.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#373

Post by roughly »

fpgaminer wrote: post #292
film42 wrote: post #262

Earlier quoted context omitted.

A thread per connection is a almost always the correct decision for performance, but by choosing a process per connection, postgres is able to let you load whatever sketchy extensions you want. Worst case you crash the process, not the database. It would be nice if you could strike a balance so a segfaul in the extension only crashes a small percentage of connections, not the whole thing.

If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.

In the age of vibe-generated code, I promise you're gonna want the safety.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#374

Post by hoppp »

fpgaminer wrote: post #292
film42 wrote: post #262

Earlier quoted context omitted.

A thread per connection is a almost always the correct decision for performance, but by choosing a process per connection, postgres is able to let you load whatever sketchy extensions you want. Worst case you crash the process, not the database. It would be nice if you could strike a balance so a segfaul in the extension only crashes a small percentage of connections, not the whole thing.

If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.

What about extensions that are not sketchy? Lots of good ones out there.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#375

Post by copperx »

IsTom wrote: post #339
rjh29 wrote: post #282

Earlier quoted context omitted.

AI is a great use for this kind of boring, rote translation where precision is important. Humans are quite bad at it and tend to make mistakes. In either case the focus should be on improving testing, not trying to manually verify if the translation was correct by eye.

With programs large enough tests aren't going to ever be enough. Formal verification might work, but then who checks the specification for bugs?

In the case of rewrites, the specification is the original behavior, no? bugs and all.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#376

Post by dmitrygr »

booksock wrote: post #245
dirkc wrote: post #78

How would one go about reviewing a piece of code like this? One of the things I'd typically do is peek at the commit history. Seeing what people worked on and how they did it tends to say a lot about a project. But with LLMs generating 7101 commits in less than a month that isn't feasible. Even looking at a single day is way too much [1]. It probably also doesn't make sense since the commits content won't tell you mu…

(I'm working with malisper on pgrust), I think the focus for projects like this is going to shift to reviewing the testing/fuzzing process instead of reviewing each commit (going much further than what the postgres regression/isolation/crash tests do). related post from danluu: https://danluu.com/ai-coding/

And what happens when your "tests" are also vibecoded. Right now all of these houses of cards reset on human-written tests. What happens without them?

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#377

Post by dmitrygr »

dirkc wrote: post #158
bakugo wrote: post #104

Earlier quoted context omitted.

Vibe code was never meant to be reviewed. These rewrites are just test-driven development taken to the absolute extreme. Created under the hope that the existing tests are exhaustive and cover every relevant use case, such that if they all pass, the rewrite must be at least as good as the original. So just go with the vibes and burn tokens until they pass, and your job is done. In practice, this is never true for any…

The challenge is that more and more people are producing project like this - 1,000s of commits and > 200k lines of code - and saying it was carefully created using agent based workflows and not vibe coded.

> The challenge is that

Why is that a challenge? As long as they are open about this, all is OK.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#378

Post by gpm »

jeltz wrote: post #323
malisper wrote: post #318

Earlier quoted context omitted.

> Threads does not offer any major performance advantage This is very not true. When it comes to parallel queries, a process model adds a ton of overhead. You can't pass pointers between processes because the address space is different. This adds a ton of overhead in a bunch of different places. For example when doing a parallel hash join, Postgres will have each worker build a local hash table. Then it will take all…

Ok ... you know PostgreSQL supports hash tables in shared memory, right? PostgreSQL could in theory share those if we wanted to. The issue is just that coding anything which uses shared memory is a lot of work. Additionally the reasons PostgreSQL does not offer Clickhouse performance has very little to do with parallelism. PostgreSQL plans to move to threading but the efforts around imporving OLAP performance are alm…

Apart from being a lot of work are you really gaining much at that point? Memory corruption can still take down both sides...

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#380

Post by hoppp »

rjh29 wrote: post #282
maxloh wrote: post #121

Earlier quoted context omitted.

For instance, the TypeScript rewrite in Go was done mostly by humans and took a year before it was released. That is how you rewrite software that people can trust.

AI is a great use for this kind of boring, rote translation where precision is important. Humans are quite bad at it and tend to make mistakes. In either case the focus should be on improving testing, not trying to manually verify if the translation was correct by eye.

I have an issue with the precision of generated code.

LLMs sometimes confidently leave things out or they will overbuild.

I use them all the time but mistakes happen. It's not exactly a scalpel, more like a sledge hammer.

Post reply on HN