Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

github.com

Post reply on HN
Print view View original

711–720 of 756 posts

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#711

Post by solid_fuel »

cowboylowrez wrote: post #589
solid_fuel wrote: post #574

Earlier quoted context omitted.

If you’re going to make a confident blanket claim, be ready to back it up - and asking for clarification is not trolling, by the way. You should be ready to engage in technical conversations if you want to make technical claims.

There is an expectation that threads are more performant than processes all else being equal. Obviously there is some work involved to make "all else be equal", like maybe threads have less isolation from each other etc but if you can mechanically convert code to use threads where it used to use processes I can see the claim standing up.

Threads generally have less memory overhead and context switching between them is theoretically faster, but as I suspect you know it is situational and not as simple as just s/process/thread/g. There are many tradeoffs, including the loss of memory isolation, and since Postgres is a network application which holds a rather important position in most architectures, proper memory isolation is very important.

Switching to threads doesn't guarantee weaker isolation, just like it doesn't guarantee better performance, but those nuances are why "switching to threading yields performance improvements" is over simplified to the point of uselessness. It simply reeks of the same kind of ignorance that used to drive statements like "rust doesn't have vulnerabilities" and "rewrite it in javascript for web scale performance".

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#712

Post by dstanko »

booksock wrote: post #244
pknerd wrote: post #44

I am not trolling, but I have a simple question: Why? Why do I use this instead of the official build? What is the business case?

(I'm working with malisper on this), we are now focusing on improving many things about postgres! Some we have written about before [0], and we have much more in mind too. Malis wrote another comment about analytical workloads being 300x faster now than postgres for a version we're working on right now Aiming for postgres compatible database with a 2026 architecture [0] https://malisper.me/the-four-horsemen-behind-th…

Why not then doing it as a fork - using existing code and language to re-architect? What value does Rust bring here? You are using LLMs to rewrite, so the language is pretty much irrelevant from developers perspective.

You can say "we want it in Rust" and leave it there - I'd be fine with it. Wouldn't use it though.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#713

Post by selfmodruntime »

cyber1 wrote: post #672
atombender wrote: post #648

Earlier quoted context omitted.

Counterpoint: All of the current Postgres codebase is already wrapped in an invisible unsafe{}. The difference with a Rust codebase like this is that all of the unsafe code has been neatly isolated and clearly marked. The outside code is safe — at least according to the definition of what Rust considers safe, which is a high bar indeed and objectively superior to the unsafe mess that is C — and the unsafe code is nat…

Valid point, but unsafe in Rust is more dangerous than unsafe in C, bcs of aliasing. For example PG is compiled with -fno-strict-aliasing.

Thank you. I write quite a lot of unsafe code myself and while safe Rust is much easier to get right than C, I'd say unsafe Rust is at least 10x harder to do correctly. Rustc's aliasing rules don't vanish when you use unsafe, you'll have to uphold them yourself!

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#714

Post by JackSlateur »

frollogaston wrote: post #364
dupontcyborg wrote: post #354

Earlier quoted context omitted.

i may be missing context, but shared memory across processes, without ipc?

Being really pedantic here, shared memory is considered IPC, but not the kind you're thinking of. Shared address space, no overhead.

"Shared address space, no overhead"

But concurrent access, so synchronization is required (lock or whatever), so overhead :)

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#715

Post by touisteur »

Might be a very good occasion to actually improve the test suites from our load-bearing software projects. I feel this will be the decade of a cat and mouse game between LLM PRs and finding good (as in convincing whomever is paying you and is waiting for any occasion to fire you for being anti-progress or something).

Hopefully we get: actual formal coding rules, spec rules, design rules, contribution rules, documentation and testing rules. High Integrity development processes impose that you write all this before you start and makes sure you follow your own rules.

So. I guess... welcome everyone to explicit software and systems development processes.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#716

Post by [deleted] »

solid_fuel wrote: post #710
rjh29 wrote: post #622

Earlier quoted context omitted.

I'm not interested in a technical conversation with you; I was answering someone else. The top voted comment thread of this page, which dominates discussion, already covers everything you'd want to know.

Ah yeah, what I figured. You clearly have no idea what you're talking about. Don't get all mad when someone calls you out on it. By the way, just so you have some concept of what the actual problem is despite your resistance to education: Simply switching from processes to threads will not yield the claimed performance increases. A 300x improvement on analytic workflows? From a direct transliteration? Your BS alarms…

[deleted]

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#717

Post by rjh29 »

solid_fuel wrote: post #710
rjh29 wrote: post #622

Earlier quoted context omitted.

I'm not interested in a technical conversation with you; I was answering someone else. The top voted comment thread of this page, which dominates discussion, already covers everything you'd want to know.

Ah yeah, what I figured. You clearly have no idea what you're talking about. Don't get all mad when someone calls you out on it. By the way, just so you have some concept of what the actual problem is despite your resistance to education: Simply switching from processes to threads will not yield the claimed performance increases. A 300x improvement on analytic workflows? From a direct transliteration? Your BS alarms…

The author was talking about eliminating memory being copied between processes. That's a performance improvement. I don't know how the 300x improvement was made or whether that's even real - you should really be arguing with the author about that. I didn't seriously put stock into it because you know, they used AI to rewrite it. I doubt their testing methodology is sound.

I appreciate the condescension though. Thanks for teaching stupid old me how a database works.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#718

Post by solid_fuel »

rjh29 wrote: post #717
solid_fuel wrote: post #710

Earlier quoted context omitted.

Ah yeah, what I figured. You clearly have no idea what you're talking about. Don't get all mad when someone calls you out on it. By the way, just so you have some concept of what the actual problem is despite your resistance to education: Simply switching from processes to threads will not yield the claimed performance increases. A 300x improvement on analytic workflows? From a direct transliteration? Your BS alarms…

The author was talking about eliminating memory being copied between processes. That's a performance improvement. I don't know how the 300x improvement was made or whether that's even real - you should really be arguing with the author about that. I didn't seriously put stock into it because you know, they used AI to rewrite it. I doubt their testing methodology is sound. I appreciate the condescension though. Thanks…

> Thanks for teaching stupid old me how a database works.

You're welcome, someone had to.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#719

Post by ahachete »

vintagedave wrote: post #195

This is impressive - but is a license change, from the PostgresQL license [0] to AGPL [1]. I like the AGPL and think it's the best truly free open source license, but I worry if this is compatible. Ie, if this is rewritten from the original source, should the original apply? (Yes.) There has been a trend to rewrite open source software with a more restrictive license (like coretools in Rust). This looks considerably…

AGPL and copyleft licenses are not restrictive, they provide forward (open source) guarantees. The only thing they "restrict" is the ability to remove freedoms. Therefore is not a restriction, is a guarantee. A guarantee that the project will endure as open source.

I wouldn't go as far as calling permissive licensing "restrictive" because they actually allow for reducing freedoms; but copyleft is definitely not restrictive at all, it's the opposite.

Having said that, I like and appreciate both kind of licenses and I have and will continue creating open source software using ones or the others.

Re: Postgres rewritten in Rust, now passing 100% of the Postgres regression tests

#720

Post by nylonstrung »

tudorg wrote: post #611
FreakLegion wrote: post #414

Earlier quoted context omitted.

pg_mooncake (now effectively abandoned due to being acquired by Databricks, but still up at https://github.com/Mooncake-Labs/pg_mooncake ) pulled the DuckDB engine into Postgres wholesale, if I remember right. pg_lake also uses DuckDB but keeps it external, routing through Postgres and managing Iceberg tables (but not the data itself) there ( https://github.com/Snowflake-Labs/pg_lake ). Both of these were neck and ne…

Actually δx is faster than the "duckdb embedded in postgres" options: https://benchmark.clickhouse.com/#system=+_etx|_b|_i )|dula|pnc&type=-&machine=-6t|ca2|6ax|g4e|6ale|3al&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=- Plus all the normal Postgres features work as expected: physical/logical replication, crash recovery, pg_dump/pg_restore, etc.

Are you using Datafusion or parts of it for query engine
Post reply on HN