Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

github.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#21

Post by q3k »

jeffbee wrote: post #14
q3k wrote: post #13

Earlier quoted context omitted.

It's 'only' really 4 (two of them are gRPC implementations). But yeah - this is one of those things that makes me stay with Go instead of moving over to Rust for my backend SOA/microservice work. In Rust, for everything you need to do, there's at least 5 different libraries that implement that, all competing with eachother. This is especially annoying when dealing with transitive dependencies. Meanwhile in Go, you ge…

There are at least two Go protoc plugins, protobuf-go and gogoproto.

It's the only real alternative implementation (and more precisely, a fork of upstream protobuf), _and_ there is strong cooperation [1] [2] between both projects to maintain a level of interoperability.

And, it's not even protobuf I have a problem with - but things like HTTP implementations. There still isn't a canonical HTTP client/server implementation for Rust, while in Go basically everyone just uses `net/http`, or something that builds on top of that. Same for cryptographic primitives, TLS, context, ...

[1] - https://docs.google.com/document/d/19kfhro7-CnBdFqFk7l4_Hmwa...

[2] - https://github.com/gogo/protobuf/issues/386

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#22

Post by haberman »

I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions.

For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing the tree of objects and calling free() on each one.

My dream has been that Rust protobuf could support arenas as well as C++, but use Rust's type system to make it all provably correct at compile time (in C++ the lifetime management is inherently manual and unsafe). For absolute top performance, arenas will always beat trees of unique pointers (which I think corresponds to Rust's Box type).

I don't know Rust's type/lifetime system well enough to know if this is possible. I was looking recently at arenas in Rust and I noticed that Rust's version of placement new seems to be stalled:

"Unfortunately the path forward for placement new in Rust does not look good right now, so I've reverted this crate to work more like a memory heap where stuff can be put, but not constructed in place."

https://docs.rs/light_arena/1.0.1/light_arena/

Does anyone know more about this?

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#23

Post by jspaetzel »

james412 wrote: post #15

I know it's common and perhaps even fashionable, but FWIW language like "We take an opinionated stance" utterly puts me off caring about this package It's a piece of software, it has a design that is either fit for purpose or not. When ego becomes entangled in that design process, it's a strong indicator of the kind of experience one might have trying to get fixes or enhancements merged, or even the kind of attitude…

More context is helpful here. "We take an opinionated stance that every module should be a crate, as opposed to generating Rust files 1:1 with proto files."

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#24

Post by rapsey »

staticassertion wrote: post #8

This is great. AFAIK this is the only protobuf library in Rust that supports zero copy. Maybe this'll help some of the other libraries implement similar features?

Quick-protobuf has been around for a while and supports it.

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#25

Post by staticassertion »

q3k wrote: post #13
deepsun wrote: post #6

There's already 6 different protobuf libraries for Rust: [1] I've chosen Prost for our project, but see the whole list: https://github.com/stepancheg/rust-protobuf#related-projects

It's 'only' really 4 (two of them are gRPC implementations). But yeah - this is one of those things that makes me stay with Go instead of moving over to Rust for my backend SOA/microservice work. In Rust, for everything you need to do, there's at least 5 different libraries that implement that, all competing with eachother. This is especially annoying when dealing with transitive dependencies. Meanwhile in Go, you ge…

> In Rust, for everything you need to do, there's at least 5 different libraries that implement that, all competing with eachother.

That's interesting. This seems like a good thing to me. I've had no issues with microservices in Rust related to having too many crates.

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#26

Post by staticassertion »

rapsey wrote: post #24
staticassertion wrote: post #8

This is great. AFAIK this is the only protobuf library in Rust that supports zero copy. Maybe this'll help some of the other libraries implement similar features?

Quick-protobuf has been around for a while and supports it.

Thanks, I hadn't seen this. I'm curious about how they compare - it looks like quick-protobuf uses Cow, but I think pb-jelly doesn't?

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#27

Post by woah »

q3k wrote: post #13
deepsun wrote: post #6

There's already 6 different protobuf libraries for Rust: [1] I've chosen Prost for our project, but see the whole list: https://github.com/stepancheg/rust-protobuf#related-projects

It's 'only' really 4 (two of them are gRPC implementations). But yeah - this is one of those things that makes me stay with Go instead of moving over to Rust for my backend SOA/microservice work. In Rust, for everything you need to do, there's at least 5 different libraries that implement that, all competing with eachother. This is especially annoying when dealing with transitive dependencies. Meanwhile in Go, you ge…

IMO the time spent choosing a library in Rust is about equal to the spent debugging a null pointer in Go, but a Rust project will have a lot less libraries than a Go project will have null pointers

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#28

Post by q3k »

jeffbee wrote: post #18
q3k wrote: post #10

Earlier quoted context omitted.

Type safety for your serialization/RPC layer.

Protobuf does not provide any type safety whatsoever. The name of the type of the message is carried in a side-channel, and the interpretation of that name is completely up to the endpoint that deserializes the message.

Once you dispatch your binary/text protobuf into a proto message type however, you do get type safety, and it makes sense to carry over that type safety to implementing languages.

Plus, Any [1] is an effort to standardize the ability to carry the proto type (as a global identifier that can be used to retrieve its schema) alongside its serialized format.

[1] - https://developers.google.com/protocol-buffers/docs/proto3#a...

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#29

Post by staticassertion »

haberman wrote: post #22

I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…

AFAIK the current method for hacking in placement new is to use something like this: https://github.com/glandium/boxext

Some more context. There used to be a `box` keyword too. https://github.com/rust-lang/rust/issues/50047

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#30

Post by jeffbee »

q3k wrote: post #21
jeffbee wrote: post #14

Earlier quoted context omitted.

There are at least two Go protoc plugins, protobuf-go and gogoproto.

It's the only real alternative implementation (and more precisely, a fork of upstream protobuf), _and_ there is strong cooperation [1] [2] between both projects to maintain a level of interoperability. And, it's not even protobuf I have a problem with - but things like HTTP implementations. There still isn't a canonical HTTP client/server implementation for Rust, while in Go basically everyone just uses `net/http`, o…

I imagine this stems from the fact that Go needed to be a complete production-ready backend language at launch, whereas Rust has other origins. At Google any "hello, world" program is a web server, at Mozilla not so much.
Post reply on HN