Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

github.com

Post reply on HN
Print view View original

461–470 of 756 posts

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#461

Post by comex »

jeltz wrote: post #323
malisper wrote: post #318

Earlier quoted context omitted.

> Threads does not offer any major performance advantage This is very not true. When it comes to parallel queries, a process model adds a ton of overhead. You can't pass pointers between processes because the address space is different. This adds a ton of overhead in a bunch of different places. For example when doing a parallel hash join, Postgres will have each worker build a local hash table. Then it will take all…

Ok ... you know PostgreSQL supports hash tables in shared memory, right? PostgreSQL could in theory share those if we wanted to. The issue is just that coding anything which uses shared memory is a lot of work. Additionally the reasons PostgreSQL does not offer Clickhouse performance has very little to do with parallelism. PostgreSQL plans to move to threading but the efforts around imporving OLAP performance are alm…

> The issue is just that coding anything which uses shared memory is a lot of work.

Doesn’t that kind of prove the parent’s point though? In theory shared memory can do anything that threads can do. But if in practice some feature doesn’t get implemented in the multi-process design (because shared memory is hard), when it likely would have been implemented in a threaded design, then that’s still an advantage for threads.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#462

Post by krupan »

I started my programming career by porting code from one model of TI calculator to another. It was code I could not have written from scratch myself at the time. I learned a lot about the two different versions of TI Basic that the calculators used, but I didn't learn how the program really worked. I can't even remember now if it was Tetris or the tank game that I ported. Maybe it was both? That was a boring English class...

I totally understand why porting code is fun. It's kind of like when I checked out drawing books from the library as a kid and just traced the pictures because my own attempts at drawing were so bad. It gives you a feeling of accomplishment, even though you didn't actually do anything that difficult. And you do learn some things along the way.

Doing the same with an LLM probably gives you that similar feeling of accomplishment, even though you didn't actually do that much (sorry, hate to say it that way). I wonder if you learn even less in the process. Maybe you just learn different things.

Now that I think about it, even writing some code from scratch with an LLM is not much different than doing a porting project. Someone else did the hard work of creating the original programs that the LLM was trained on, and now you (the LLM really) are just porting/restating what someone else did. I hadn't thought of that before

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#463

Post by skybrian »

film42 wrote: post #262
malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

A thread per connection is a almost always the correct decision for performance, but by choosing a process per connection, postgres is able to let you load whatever sketchy extensions you want. Worst case you crash the process, not the database. It would be nice if you could strike a balance so a segfaul in the extension only crashes a small percentage of connections, not the whole thing.

A more modern way to do this might be to support WebAssembly plugins.

The extensions might need to be rewritten, but hey, we have AI for that now, so why not :-)

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#464

Post by moi2388 »

processunknown wrote: post #360
____tom____ wrote: post #356

Earlier quoted context omitted.

Yeah. I'm gonna have an LLM rewrite Star Wars and then film it. Should be fine, right?

LGTM. copyright laws don't matter anymore.

The only good thing to come out of AI.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#466

Post by anonzzzies »

sdevonoes wrote: post #251

Don’t understand these rewrites. - typically they are behind a single person. That’s usually bad because of spf - typically they are achieved in a very short amount of time, so the author hasn’t acquired any discipline in creating the project. That means it’s unlikely the author is going to stick to the project in the mid and long term - anyone that wants to contribute to the project needs to pay. Needs to pay tokens…

It's not just a rewrite ; it has improvements. I did the same thing for fun for the same reason; I wanted to see if I can improvements on some of the legacy design stuff and, especially, the stuff PG people have told us that it cannot be done differently. It can. I would not put it in production, but it thought me a lot about the internals of databases. To keep my brain happy in the age of LLMs, I implement database things on our (also old but many times refactored/rewritten) production db without an LLM. I'm sweating through Flexible Paxos now; probably we will just keep using raft as it's old and stable and simple but it's interesting anyway.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#467

Post by LtWorf »

fragmede wrote: post #325
LtWorf wrote: post #301

Earlier quoted context omitted.

Rust doesn't have ACID and I'm sure this doesn't either. I'd like to know if the "authors" know what I'm talking about.

You think that's bad, MongoDB doesn't even have schemas. Schemas!

Oh we used it in my first job. It was really slow and the data in there was all buggy.

We really should have used postgres instead, but it was the "big data" period so you had to use mongodb.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#468

Post by sqquima »

pier25 wrote: post #407
sdevonoes wrote: post #251

Don’t understand these rewrites. - typically they are behind a single person. That’s usually bad because of spf - typically they are achieved in a very short amount of time, so the author hasn’t acquired any discipline in creating the project. That means it’s unlikely the author is going to stick to the project in the mid and long term - anyone that wants to contribute to the project needs to pay. Needs to pay tokens…

> it’s increasingly difficult to maintain these projects without AI It's pretty much impossible in a project of this size. IIRC Postgres has over 1M loc.

Are there modular databases with clearly differentiated components? I wouldn't be surprised if there aren't, as the trade-off with modularity is usually performance.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#469

Post by f311a »

levkk wrote: post #296
malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

Ouf. I don't know. I don't want to call you out without evidence -- I myself make benchmark claims all the time -- but 50% improvement in OLTP seems suspicious. I get that you used a standard benchmark, and I don't even know what it entails, but my spidey sense is going off. Perhaps, some trade off somewhere that won't make it to prod because it breaks MVCC -- and yes, I saw that it passes regression tests. Just chec…

Yeah, claims that it can be faster than CH are very suspicious. CH guys are very good at their craft, they spend hundreds of hours optimizing one single small detail.
Post reply on HN