Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

github.com

Post reply on HN
Print view View original

311–320 of 756 posts

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#311

Post by booksock »

solid_fuel wrote: post #280
booksock wrote: post #244

Earlier quoted context omitted.

(I'm working with malisper on this), we are now focusing on improving many things about postgres! Some we have written about before [0], and we have much more in mind too. Malis wrote another comment about analytical workloads being 300x faster now than postgres for a version we're working on right now Aiming for postgres compatible database with a 2026 architecture [0] https://malisper.me/the-four-horsemen-behind-th…

> Aiming for postgres compatible database with a 2026 architecture Except you didn't improve the architecture, did you? You just asked an LLM to copy what was already there. Making real improvements to the database architecture requires understanding the database architecture, not just asking a calculator to do the work for you. Better benchmark performance means nothing if the underlying guarantees break, and a 300x…

The version we have live right now is pre architecture changes, we wanted to make sure we could hit this milestone first. And agree about proving the underlying guarantees. It will be pretty exciting when we do

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#312

Post by rsoto2 »

sshine wrote: post #14
gingersnap wrote: post #3

I start to see a lot of these re-writes that depend on tests to state that its working. But the things that make software like Postgres and SQLite reliable are not mostly the test, but the real world production scars. That's where the reliability comes from, years and years of running in production.

> not mostly the test, but the real world production scars Most extensive test suites are exactly production scars: every time you have a bug or a regression, you write a test that confirms correct behaviour. SQLite is a good example to bring up because its extensive closed-source tests are what’s often cited as being what keeps people from forking it. (Turso did it, though, but it takes a company to deliver some gua…

having a large test suite does not equal to testing every potential edge scenario that has never broken before in production.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#313

Post by jeltz »

codesnik wrote: post #308
jeltz wrote: post #303

Earlier quoted context omitted.

Threads does not offer any major performance advantage, performance of processes vs threads is virtually the same. The reason the PostgreSQL project is moving towards threads is to make development easier.

unless you're spawning them for new connections.

Some, but not that much. Switching PostgreSQL to a threaded model will not magically make spawning connections fast. PostgreSQL connections are quite heavyweight.

The reason to use threads is almost entirely about ease of development, not about performance. If you use shrared memory like PostgreSQL does you need to write your own allocators, etc. So much you get for free if you use threads.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#314

Post by queenkjuul »

subygan wrote: post #286
maxloh wrote: post #121

Earlier quoted context omitted.

For instance, the TypeScript rewrite in Go was done mostly by humans and took a year before it was released. That is how you rewrite software that people can trust.

What about human written software makes it more reliable than LLM written software? is it the craftsmanship, or the deliberate decision making of industry veterans?

It's the notably poor quality of LLM-generated code

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#315

Post by dcrazy »

wrs wrote: post #250
booksock wrote: post #245

Earlier quoted context omitted.

(I'm working with malisper on pgrust), I think the focus for projects like this is going to shift to reviewing the testing/fuzzing process instead of reviewing each commit (going much further than what the postgres regression/isolation/crash tests do). related post from danluu: https://danluu.com/ai-coding/

Some of this post reminds me of a story I heard long ago from someone who had worked at a HW/SW company. They’d transferred an engineer from the ASIC design team to the OS kernel team, though he’d never been on a software team before. After a while the manager called him in for the following conversation: Manager: You’re doing amazing work — zero bugs in production! I’d like you to mentor the other SWEs on how to get…

Hardware engineers call them errata ;-)

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#316

Post by jeltz »

wrs wrote: post #250
booksock wrote: post #245

Earlier quoted context omitted.

(I'm working with malisper on pgrust), I think the focus for projects like this is going to shift to reviewing the testing/fuzzing process instead of reviewing each commit (going much further than what the postgres regression/isolation/crash tests do). related post from danluu: https://danluu.com/ai-coding/

Some of this post reminds me of a story I heard long ago from someone who had worked at a HW/SW company. They’d transferred an engineer from the ASIC design team to the OS kernel team, though he’d never been on a software team before. After a while the manager called him in for the following conversation: Manager: You’re doing amazing work — zero bugs in production! I’d like you to mentor the other SWEs on how to get…

Funny story but in my experience hardware engineers produce some of the worst software of the industry. Of course there must be some hardware engineers out there who do hood software but generally what they build are disasters.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#317

Post by queenkjuul »

satvikpendem wrote: post #12
juliangmp wrote: post #2

I feel like we need to heavily differentiate between a rewrite and an AI rewrite.

It is more and more the future. No human would want to rewrite one technology to another because it is too marginal a gain. AI on the other hand does not give a shit.

> No human would want to rewrite one technology to another

Except for when they do, like the new TypeScript...

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#318

Post by malisper »

jeltz wrote: post #303
fpgaminer wrote: post #292

Earlier quoted context omitted.

If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.

Threads does not offer any major performance advantage, performance of processes vs threads is virtually the same. The reason the PostgreSQL project is moving towards threads is to make development easier.

> Threads does not offer any major performance advantage

This is very not true. When it comes to parallel queries, a process model adds a ton of overhead. You can't pass pointers between processes because the address space is different. This adds a ton of overhead in a bunch of different places. For example when doing a parallel hash join, Postgres will have each worker build a local hash table. Then it will take all the tuples out of the local hash table and copy them through shared memory to the leader who will then construct a new hash table. This duplicates a lot of work as you have to hash the tuples multiple times.

A lot of getting to Clickhouse level performance was making better use of parallelism.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#319

Post by jeltz »

Suzuran wrote: post #241
bbg2401 wrote: post #228

Earlier quoted context omitted.

> If a Rust rewrite of any of your software becomes available and you aren't installing it immediately and without reservation This is silly. Rust is awesome, and it's hard to argue against in many domains. However, software is more than the language it is written in or the runtime serving it. Is the Rust rewrite fully compatible? Is it supported by a strong community? Is it likely to continue to be supported? Is its…

None of those concerns approach the level of priority that must be assigned to security. On defense, your security must be perfect forever or you are absolutely defeated. None of us are on the red team. It's not a rash decision, it's the only decision that logic allows. If you are not secure, you are NOTHING.

I strongly disagree. The easiest way to shut down your business is to insist on being 100% secure because the only way to have perfect security is to do nothing.

Security is always about tradeoffs.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#320

Post by fpgaminer »

booksock wrote: post #245
dirkc wrote: post #78

How would one go about reviewing a piece of code like this? One of the things I'd typically do is peek at the commit history. Seeing what people worked on and how they did it tends to say a lot about a project. But with LLMs generating 7101 commits in less than a month that isn't feasible. Even looking at a single day is way too much [1]. It probably also doesn't make sense since the commits content won't tell you mu…

(I'm working with malisper on pgrust), I think the focus for projects like this is going to shift to reviewing the testing/fuzzing process instead of reviewing each commit (going much further than what the postgres regression/isolation/crash tests do). related post from danluu: https://danluu.com/ai-coding/

For large projects like this I think a hierarchical division of labor also helps.

If you first carefully define the overall architecture and thus individual high level components of the system, then you know which of those components are mission critical and which are commodity. Mission critical would be anything ensuring ACID, etc. That way, no matter what you farm out to LLMs, you can keep the majority of limited human focus on the far fewer mission critical components. If tests end up not being robust enough to catch all issues, at least they'll be isolated to commodity code where damage is limited to things like DoS, etc, and not code that could cause data loss.

I also think it's important to first define the _contracts_ on and between each of these components, and derive tests from those contracts. Partly because contracts more succinct and easier to reason about. And partly because Rust provides many tools to enforce contracts at compile time, reducing the need for tests (which themselves could end up subtly flawed). Contracts can be enforced through typing, private vs public APIs, etc. Newtypes are _incredibly_ powerful for both enforcing contracts and making footguns much less likely.

Post reply on HN