Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

github.com

Post reply on HN
Print view View original

291–300 of 756 posts

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#291

Post by OsrsNeedsf2P »

malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

What's your actual background and expertise with Postgres and databases more broadly? Basically, do you actually know what you're doing, or is there likely a massive footgun you don't know or haven't shared with us?

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#292

Post by fpgaminer »

film42 wrote: post #262
malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

A thread per connection is a almost always the correct decision for performance, but by choosing a process per connection, postgres is able to let you load whatever sketchy extensions you want. Worst case you crash the process, not the database. It would be nice if you could strike a balance so a segfaul in the extension only crashes a small percentage of connections, not the whole thing.

If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#294

Post by happyPersonR »

malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

Do you have anything in the regression test suite like jepsen etc?

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#295

Post by fluffybucktsnek »

film42 wrote: post #262
malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

A thread per connection is a almost always the correct decision for performance, but by choosing a process per connection, postgres is able to let you load whatever sketchy extensions you want. Worst case you crash the process, not the database. It would be nice if you could strike a balance so a segfaul in the extension only crashes a small percentage of connections, not the whole thing.

I'm not informed of the Postgres's internals, but, maybe, that can be solved by grouping threads into different processes depending on which set of extensions they request.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#296

Post by levkk »

malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

Ouf. I don't know. I don't want to call you out without evidence -- I myself make benchmark claims all the time -- but 50% improvement in OLTP seems suspicious. I get that you used a standard benchmark, and I don't even know what it entails, but my spidey sense is going off. Perhaps, some trade off somewhere that won't make it to prod because it breaks MVCC -- and yes, I saw that it passes regression tests.

Just checking, is fsync on? :) Regression tests don't catch bad IO patterns afaik.

Anyway... sounds like a fun project to work on!

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#297

Post by LtWorf »

levkk wrote: post #296
malisper wrote: post #147

Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…

Ouf. I don't know. I don't want to call you out without evidence -- I myself make benchmark claims all the time -- but 50% improvement in OLTP seems suspicious. I get that you used a standard benchmark, and I don't even know what it entails, but my spidey sense is going off. Perhaps, some trade off somewhere that won't make it to prod because it breaks MVCC -- and yes, I saw that it passes regression tests. Just chec…

Remember when databases were faster to run in virtualbox rather than bare metal? (because virtual box was completely ignoring all the instructions to flush the data on the disks)

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#299

Post by solomatov »

malisper wrote: post #270
solomatov wrote: post #247

Earlier quoted context omitted.

Super impressive! Is it possible for you to share your methodology of using LLMs?

My approach has changed throughout the course of this project. Throughout most of the project, we were working off of a c2rust translation of Postgres to Rust. That gave us a bunch of Rust code that was unsafe but did pass the Postgres test suite and was fast. c2rust had split Postgres into 1000 different crates. We then went through 1 by 1 and rewrote each crate into idiomatic rust. This naturally lended itself to a…

Thanks for sharing.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#300

Post by [deleted] »

fluffybucktsnek wrote: post #295
film42 wrote: post #262

Earlier quoted context omitted.

A thread per connection is a almost always the correct decision for performance, but by choosing a process per connection, postgres is able to let you load whatever sketchy extensions you want. Worst case you crash the process, not the database. It would be nice if you could strike a balance so a segfaul in the extension only crashes a small percentage of connections, not the whole thing.

I'm not informed of the Postgres's internals, but, maybe, that can be solved by grouping threads into different processes depending on which set of extensions they request.

[deleted]
Post reply on HN