Great work Lev!
Thank you!
Replacing Protobuf with Rust
91–100 of 135 posts
Re: Replacing Protobuf with Rust
#92Gotta say, I love using PGDog. It has some fantastic built in features, and I'm looking forward to testing out the improved query parser. Lev and the team are heroes. At the scale we were using PGDog, enabling the previous form of the query parser was extremely expensive (we would have had to 16x our pgdog fleet size).
That's the experimental feature I was talking about! :)
Thank you so much for the kind words!
Re: Replacing Protobuf with Rust
#93Performance of Protobuf is a joke. Why not use a zero copy format so that serialization is free? For example, my format Lite³ which outperforms Google Flatbuffers by 242x: https://github.com/fastserial/lite3
Re: Replacing Protobuf with Rust
#94You should be terrified of the instability you're introducing to achieve this. Memory sharing between processes is very difficult to keep stable, it is half the reason kernels exist.
I was terrified until it worked. The Postgres "ABI" is relatively stable - the parser only really changes between major versions and we bake the whole code into the same executable - largely thanks to the work done by team behind pg_query!
The output is machine-verifiable, which makes this uniquely possible in today's vibe-coded world!
Re: Replacing Protobuf with Rust
#95Earlier quoted context omitted.
No it’s not common for two pieces of code within a single process to communicate by serializing the protobuf into the wire format and deserializing it. It’s however somewhat common to pass in-memory protobuf objects between code, because the author didn’t want to define a custom struct but preferred to use an existing protobuf definition.
I agree it's not super common, but Boq's in-process RPC feature encourages this pattern.
Except it is not a remote procedure call.
Re: Replacing Protobuf with Rust
#96Just for fun, how often do regular-sized companies that deal in regular-sized traffic need Protobuf to accomplish their goals in the first place, compared to JSON or even XML with basic string marshalling?
I dunno, are you sure you can manually write correct de/serializaiton for JSON and XML so strings, floats and integer formats correctly get parsed between JavaScript, Java, Python, Go, Rust, C++ and any other languages? Do you want to maintain that and debug that? Do you want to do all of that without help of a compiler enforcing the schema and failing compiles/CI when someone accidentally changes the schema? Because…
I mean, the entire internet has been doing that for decades and there’s a lot of tooling, libraries and generators that already do that, so … sure?
And it works in a browser, too!
Re: Replacing Protobuf with Rust
#97What I find particularly ironic is that the title make it feel like Rust gives a 5x performance improvement when it actually slows thing down. The problem they have software written in Rust, and they need to use the libpg_query library, that is written in C. Because they can't use the C library directly, they had to use a Rust-to-C binding library, that uses Protobuf for portability reasons. Problem is that it is slo…
> What I find particularly ironic is that the title make it feel like Rust gives a 5x performance improvement when it actually slows thing down.
Rust didn't slow them down. The inefficient design of the external library did.
Calling into C libraries from Rust is extremely easy. It takes some work to create a safer wrapper around C libraries, but it's been done for many popular libraries.
This is the first and only time I've seen an external library connected via a Rube Goldberg like contraption with protobufs in the middle. That's the problem.
Sadly they went with the "rewrite to Rust" meme in the headline for more clickability.
Re: Replacing Protobuf with Rust
#98Just for fun, how often do regular-sized companies that deal in regular-sized traffic need Protobuf to accomplish their goals in the first place, compared to JSON or even XML with basic string marshalling?
Besides the other comments already here about code gen & contracts, a bigger one for me to step away from json/xml is binary serialization. It sounds weird, and its totally dependent on your use case, but binary serialization can make a giant difference. For me, I work with 3D data which is primarily (but not only) tightly packed arrays of floats & ints. I have a bunch of options available: 1. JSON/XML, readable, eas…
This use case totally makes sense of course. I’m thinking about why people use Protobuf for their string, uuid and int powered CRUD app.
Re: Replacing Protobuf with Rust
#99Just for fun, how often do regular-sized companies that deal in regular-sized traffic need Protobuf to accomplish their goals in the first place, compared to JSON or even XML with basic string marshalling?
In most languages protobuf is eaiser because it generates the boilerplate. And protobuf is cross language so even if you are working in javascript where json is native protobuf is still faster because the other side can be whatever and you are not spending their time parsing.
In most languages I’ve worked in, there is no boiler plate for json either, and barely any for XML. You make a data class of some sort and it “just works”.
Not having that functionality is a weakness of a language or its support tools at this point, to me.
Re: Replacing Protobuf with Rust
#100Earlier quoted context omitted.
You're saying we choose Protobufs [1] because Google maintains it but not FlatBuffers [2]? [1] - https://github.com/protocolbuffers/protobuf : Google's data interchange format [2] - https://github.com/google/flatbuffers : Also maintained by Google
I get the OP is off base with his remark - but at the same time maintained by Google means shit in practice. AFAIK they have a bunch of production infra on protobuff/gRPC - not so sure about flatbufferrs which came out of the game dev side - that's the difference maker to me - which project is actually rooted in.
> AFAIK they have a bunch of production infra on protobuff/gRPC
Stubby, not gRPC. Stubby is used for almost everything internally. gRPC is a similar-ish looking thing that is open sourced, but not used nearly as much as stubby internally.
Stubby predates gRPC by like 15 years or something.
> not so sure about flatbufferrs which came out of the game dev side
I wouldn't know. I'll be honest, I always forget that Google made flatbuffers. I guess if you're doing a lot of IPC?