Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replacing Protobuf with Rust

pgdog.dev

Post reply on HN
Print view View original

111–120 of 135 posts

Re: Replacing Protobuf with Rust

#111

Post by t-writescode »

tucnak wrote: post #102
t-writescode wrote: post #98

Earlier quoted context omitted.

This use case totally makes sense of course. I’m thinking about why people use Protobuf for their string, uuid and int powered CRUD app.

You're making assumptions about what kind of software people write. For a Hacker News degenerate, everything in the world revolves around bean-counting B2B SaaS CRUD crap, but it doesn't mean it's all there is to the world, right? You would be shocked how much networked computer software (not everything is a website) exists that is NOT a CRUD "app."

Woah buddy, no need for the hostility there.

Statistically, a lot of people who post on HN and cling to new or advanced tech *do* just write CRUD apps with a little special sauce, it’s part of what makes vibe coding and many of the frameworks we use so appealing.

I’m not ignoring that other things exist and are even very common; and I was agreeing with the person that’s a useful case.

I’ve also worked for various companies where protobuf has been suggested as a way to solve a political/organizational issue, not a code or platform issue.

Re: Replacing Protobuf with Rust

#112

Post by driftwood4537 »

Aurornis wrote: post #97
GuB-42 wrote: post #57

What I find particularly ironic is that the title make it feel like Rust gives a 5x performance improvement when it actually slows thing down. The problem they have software written in Rust, and they need to use the libpg_query library, that is written in C. Because they can't use the C library directly, they had to use a Rust-to-C binding library, that uses Protobuf for portability reasons. Problem is that it is slo…

> What I find particularly ironic is that the title make it feel like Rust gives a 5x performance improvement when it actually slows thing down. Rust didn't slow them down. The inefficient design of the external library did. Calling into C libraries from Rust is extremely easy. It takes some work to create a safer wrapper around C libraries, but it's been done for many popular libraries. This is the first and only ti…

What about the other way around? i recently had a use case where i needed a C shared library that persists complex C data structures into an RDBMS. Given my team had minimal C experience and this needed to be production grade. I ended up writing a thin C lib that offloads the heavy lifting to a sidecar go process. They interacted via protobuf over a local unix socket.

Would love to hear if i could've come up with a better design.

Re: Replacing Protobuf with Rust

#113

Post by rurban »

yodacola wrote: post #9

FlatBuffers are already faster than that. But that's not why we choose Protobuf. It's because a megacorp maintains it.

I also thought I could trust mega Corp. That's why I put all my code on their platform, code.google.com, and not on this obscure platform without any business model, github. Well, that sucked. And why should I use protobuf, when I just need to share structs and arrays in memory (aka zero copy) with a version field? Like everyone else does for decades?

Re: Replacing Protobuf with Rust

#114

Post by lfittl »

Since there seems to be some confusion in the comments about why pg_query chose Protobufs in the first place, let me add some context as the original author of pg_query (but not involved with PgDog, though Lev has shared this work by email beforehand).

The initial motivation for developing pg_query was for pganalyze, where we use it to parse queries extracted from Postgres, to find the referenced tables, and these days also rewrite and format queries. That use case runs in the background, and as such is much less performance critical.

pg_query actually initially used a JSON format for the parse output (AST), but we changed that to Protobuf a few major releases ago, because Protobuf makes it easy to have typed bindings in the different languages we support (Ruby, Go, Rust, Python, etc). Alternatives (e.g. using FFI directly) make sense for Rust, but would require a lot of maintained glue code for other languages.

All that said, I'm supportive of Lev's effort here, and we'll add some additional functions (see [0]) in the libpg_query library to make using it directly (i.e. via FFI) easier. But I don't see Protobuf going away, because in non-performance critical cases, it is more ergonomic across the different bindings.

[0]: https://github.com/pganalyze/libpg_query/pull/321

Re: Replacing Protobuf with Rust

#115

Post by DangitBobby »

izacus wrote: post #46
desiderantes wrote: post #14

Earlier quoted context omitted.

That never happens. Instead, it always attracts the opposite group, the Rust complainers, where they go and complain about how "the everything-in-rust-is-better crowd created yet another fake headline to pretend that Rust is the panacea". Which results in a lot of engagement. Old ragebait trick.

"never" huh?

Pretty much. The tide of rust evangelism has been turned in favor of complainers for a while now. Nothing compared to JS and React hate, but still.

Re: Replacing Protobuf with Rust

#116

Post by jpalepu33 »

The title is misleading but the actual work is impressive - they optimized their Protobuf usage, not replaced it entirely.

This is a common pattern: "We switched to X and got 5x faster" often really means "We fixed our terrible implementation and happened to rewrite it in X."

Key lessons from this:

1. Serialization/deserialization is often a hidden bottleneck, especially in microservices where you're doing it constantly 2. The default implementation of any library is rarely optimal for your specific use case 3. Benchmarking before optimization is critical - they identified the actual bottleneck instead of guessing

For anyone dealing with Protobuf performance issues, before rewriting: - Use arena allocation to reduce memory allocations - Pool your message objects - Consider if you actually need all the fields you're serializing - Profile the actual hot path

Rust FFI has overhead too. The real win here was probably rethinking their data flow and doing the optimization work, not just the language choice.

Re: Replacing Protobuf with Rust

#118

Post by wizzwizz4 »

GuB-42 wrote: post #108
Aurornis wrote: post #97

Earlier quoted context omitted.

> What I find particularly ironic is that the title make it feel like Rust gives a 5x performance improvement when it actually slows thing down. Rust didn't slow them down. The inefficient design of the external library did. Calling into C libraries from Rust is extremely easy. It takes some work to create a safer wrapper around C libraries, but it's been done for many popular libraries. This is the first and only ti…

> Calling into C libraries from Rust is extremely easy Calling the C function is not the problem here. It is dealing with the big data structure this function returns in a Rust-friendly manner. This is something Protobuf does very well, at the cost of performance.

Writing Rust bindings for arbitrary C data structures is not hard. You just need to make sure every part of your safe Rust API code upholds the necessary invariants. (Sometimes that's non-trivial, but a little thinking will always yield a solution: if C code can do it, then it can be done, and if it can be done, then it can be done in Rust.)

Re: Replacing Protobuf with Rust

#119

Post by ajross »

Seems like this has nothing to do with Rust or protobufs. The underlying PostgreSQL abstraction engine they'd picked had a wasteful serialization implementation (that happens to have been using protobuf). So pgdog dropped it and open-coded a serialization-free transfer using the C API.

Well, yeah. If there's a feature you don't need, you'll see value by coding around it. Some features turn out not to be needed by anyone, maybe this is one. But some people need serialization, and that's what protobufs are for[1]. Those people are very (!) poorly served by headlines telling them to use Rust (!!) instead of serialization.

[1] Though as always the standard litany applies: you actually want JSON, and not protobus or ASN.1 or anything else. If you like some other technology better, you're wrong and you actually want JSON. If you think you need something faster, you probably don't and JSON would suit your needs better. If you really, 100%, know for sure that you need it faster than JSON, then you're probably isomorphic to the folks in the linked article, shouldn't have been serializing at all, and should get to work open coding your own hooks on the raw backend.

Re: Replacing Protobuf with Rust

#120

Post by satvikpendem »

MrDarcy wrote: post #47
cranx wrote: post #32

I find the title a bit misleading. I think it should be titled It’s Faster to Copy Memory Directly than Send a Protobuf. Which then seems rather obvious that removing a serialization and deserialization step reduces runtime.

TIL serializing a protobuf is only 5 times slower than copying memory, which is way faster than I thought it’d be. Impressive given all the other nice things protobuf offers to development teams.

5x is pretty slow honestly. Imagine anything happening 5x as slow as you'd expect it to. I mean, for a recent project I had to inline Rust structs rather than parsing JSON too for specific fields, and that definitely sped it up.
Post reply on HN