Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-copy protobuf and ConnectRPC for Rust

medium.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Zero-copy protobuf and ConnectRPC for Rust

#41

Post by sa46 »

codedokode wrote: post #36
nu11ptr wrote: post #34

I'd like to use this, but I don't want to refactor all my services when they change the request/response types. Interested to know the timing of 1.x. It seems to be moving pretty fast atm - hopefully that momentum keeps going.

As I understand, protobuf has compatibility (it stores field ids), so new service can read request from older client, and vice versa, so you do not need to refactor anything. Also, it is made for long-range communications, and is inefficient for inter-process or inter-thread messaging.

Presumably, OP refers to the generated rust types which depend on the specific protobuf framework.

I had the same issue when looking to adopt ConnectRPC for Go, which uses a custom wrapper type to model requests.

Re: Zero-copy protobuf and ConnectRPC for Rust

#42

Post by nly »

nopurpose wrote: post #22
secondcoming wrote: post #16

Earlier quoted context omitted.

Is this still true? New versions of protobuf allow codegen of `std::string_view` rather than `const std::string&` (which forces a copy) of `string` and `repeated byte` fields. https://protobuf.dev/reference/cpp/string-view/

It allows avoiding allocations, but it doesn't allow using serialised data as a backing memory for an in-language type. Protobuf varints have to be decoded and written out somewhere. They cannot be lazily decoded efficiently either: order of fields in the serialised message is unspecified, hence it either need to iterate message over and over finding one on demand or build a map of offsets, which negates any wins zer…

The win could be only decoding the fields you actually care about, rather than all fields.

It's the same for any other high performance decoding of TLV formats (FIX in finance for instance).

Re: Zero-copy protobuf and ConnectRPC for Rust

#43

Post by bluGill »

codedokode wrote: post #38
bluGill wrote: post #30

Earlier quoted context omitted.

Until you have to work with big and little endian systems. There are other weirdness about how different computers represent things as well. utf-8 / ucs-16 strings (or other code pages). Not all floats are ieee-754. Still when you can ignore all those issues what you did is really easy and often works.

I disagree. Big endian is long dead and not worth worrying about. And code pages too. What is more important, is dealing with schema changes, when you add new fields to requests and responses.

There are niches where those matter.

but yes schema changes is most likely to get you today

Re: Zero-copy protobuf and ConnectRPC for Rust

#44

Post by mgaunard »

codedokode wrote: post #39
pjc50 wrote: post #28

Earlier quoted context omitted.

Provided that: - you agree never to care about endianness (can probably get away with this now) - you don't want to represent anything complicated or variable length, including strings

You can have strings by using relative pointers ("string starts 123 bytes before this").

You can also just use an array which sets a max capacity, and either use a null-terminator or a separate size field.

In practice you probably want to have both, and choose what's most practical based on the message.

Re: Zero-copy protobuf and ConnectRPC for Rust

#45

Post by mgaunard »

codedokode wrote: post #37
mgaunard wrote: post #18

It's 2026 and I'm still defining my own messaging and wire protocols. Plain C structs that fit in a UDP datagram that you can reinterpret_cast from is still best. You can still provide schemas and UUIDs for that, and dynamically transcode to JSON or whatever.

What you use is perfect for short-range communication (application and child process talking over shared memory), but not good for long-range communication (over Internet) because you can have old client talking to new version of a server, so you will have to add version numbers and have the code to parse outdated formats. But protobuf has compatibility built in and you do not need to write anything to support outdat…

I already said you can UUIDs and schemas, and even dynamic conversion between mismatched schemas.

Doing plain C structs doesn't prevent any of this.

Re: Zero-copy protobuf and ConnectRPC for Rust

#46

Post by codedokode »

mgaunard wrote: post #45
codedokode wrote: post #37

Earlier quoted context omitted.

What you use is perfect for short-range communication (application and child process talking over shared memory), but not good for long-range communication (over Internet) because you can have old client talking to new version of a server, so you will have to add version numbers and have the code to parse outdated formats. But protobuf has compatibility built in and you do not need to write anything to support outdat…

I already said you can UUIDs and schemas, and even dynamic conversion between mismatched schemas. Doing plain C structs doesn't prevent any of this.

It requires extra effort to write conversion algorithm for older data structure version.

Re: Zero-copy protobuf and ConnectRPC for Rust

#47

Post by mgaunard »

codedokode wrote: post #46
mgaunard wrote: post #45

Earlier quoted context omitted.

I already said you can UUIDs and schemas, and even dynamic conversion between mismatched schemas. Doing plain C structs doesn't prevent any of this.

It requires extra effort to write conversion algorithm for older data structure version.

The converter is generated automatically based on the differences between the two schemas.

Takes zero effort other than CPU cycles.

Post reply on HN