Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

github.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#41

Post by [deleted] »

staticassertion wrote: post #36
zozbot234 wrote: post #35

Earlier quoted context omitted.

Rust objects can always be moved to arbitrary places in memory, so I'm not sure how "in place" construction of a Rust object can even make sense unless it explicitly involves the Pin feature to prevent moves. This is quite different from C/C++ where there's no default expectation that a constructed object can be bitwise-copied somewhere else in memory. Even Box is really a special case since the object is meant to be…

Placement new is just avoiding a copy. Without placement new allocating an object involves: a) Constructing the object on the stack b) Copying it to the heap With placement new you construct the object directly on the heap, thus avoiding a copy. It's purely an optimization to avoid a single copy. It's a bit surprising that Rust doesn't have it, given the focus on control over such things, and that it had an experimen…

[deleted]

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#42

Post by nemothekid »

staticassertion wrote: post #36
zozbot234 wrote: post #35

Earlier quoted context omitted.

Rust objects can always be moved to arbitrary places in memory, so I'm not sure how "in place" construction of a Rust object can even make sense unless it explicitly involves the Pin feature to prevent moves. This is quite different from C/C++ where there's no default expectation that a constructed object can be bitwise-copied somewhere else in memory. Even Box is really a special case since the object is meant to be…

Placement new is just avoiding a copy. Without placement new allocating an object involves: a) Constructing the object on the stack b) Copying it to the heap With placement new you construct the object directly on the heap, thus avoiding a copy. It's purely an optimization to avoid a single copy. It's a bit surprising that Rust doesn't have it, given the focus on control over such things, and that it had an experimen…

IIRC, Rust did have this optimization (or LLVM was doing it), but then there was a version bump or change in IR that disabled it and it wasn't trivial to fix.

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#43

Post by staticassertion »

nemothekid wrote: post #42
staticassertion wrote: post #36

Earlier quoted context omitted.

Placement new is just avoiding a copy. Without placement new allocating an object involves: a) Constructing the object on the stack b) Copying it to the heap With placement new you construct the object directly on the heap, thus avoiding a copy. It's purely an optimization to avoid a single copy. It's a bit surprising that Rust doesn't have it, given the focus on control over such things, and that it had an experimen…

IIRC, Rust did have this optimization (or LLVM was doing it), but then there was a version bump or change in IR that disabled it and it wasn't trivial to fix.

There should be an optimization pass for this, certainly, but the placement new syntax guarantees it.

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#44

Post by kevincox »

staticassertion wrote: post #36
zozbot234 wrote: post #35

Earlier quoted context omitted.

Rust objects can always be moved to arbitrary places in memory, so I'm not sure how "in place" construction of a Rust object can even make sense unless it explicitly involves the Pin feature to prevent moves. This is quite different from C/C++ where there's no default expectation that a constructed object can be bitwise-copied somewhere else in memory. Even Box is really a special case since the object is meant to be…

Placement new is just avoiding a copy. Without placement new allocating an object involves: a) Constructing the object on the stack b) Copying it to the heap With placement new you construct the object directly on the heap, thus avoiding a copy. It's purely an optimization to avoid a single copy. It's a bit surprising that Rust doesn't have it, given the focus on control over such things, and that it had an experimen…

I haven't checked this case but often the compiler will elide the copy. However for cases like this you want it to be very reliable to guaranteed so it would be nice to get that support.

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#45

Post by ajkjk »

james412 wrote: post #20
ajkjk wrote: post #17

Earlier quoted context omitted.

That's not what the word 'opinionated' means here. It's not any one person's opinion; it's that the project overall takes a stance on an issue rather than leaving everything open for everyone else to figure out. It provides clarity and direction compared to the more difficult situation where every library is completely general. No ego involved at all.

Perhaps I misunderstand the text in the README. Who is "we" in this case? Is the software writing its own README?

'We' refers to the authors and their organization as a collective. That's still the meaning of the word 'opinionated' in this context.

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#46

Post by zamalek »

haberman wrote: post #22

I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…

Check out bumpalo[1]. Dodrio is a real-world[2] usage of bumpalo, so you can inspect that code to see how he deals with lifetimes correctly.

Edit: GATs are how "placement new" (Rust doesn't have new at all, hence the airquotes) would work in Rust, assuming the author here meant to say "custom allocators." With GATs, you could create a pointer (Box, Arc, ArenaBox etc.) trait and use that on your message types.[3] "Placement new" is a whole different issue that ultimately boils down to a compiler optimization that is currently missing/not working and shouldn't functionally affect what you are trying to do at all.

[1]: https://github.com/fitzgen/bumpalo [2]: https://github.com/fitzgen/dodrio [3]: https://rust-lang.github.io/rfcs/1598-generic_associated_typ...

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#47

Post by SOLAR_FIELDS »

foolfoolz wrote: post #19
jl2718 wrote: post #9

Is there really any good reason for code-gen? Just because “google did it” doesn’t mean it’s a good idea.

code generation is great for sharing data models and writing clients for services without dependencies

Exactly this. Imagine you work at BigCorp and use protobuf to pass data around - now you have a unified data model you can share and everyone can use the same client to access it without going through the trouble of maintaining all those getters and setters. Rolling your own getters and setters is fine in a small project but you really see the advantages of the code gen approach once you are dealing with multiple different teams in an org working with the same complex data model.

There are definitely some downsides to the approach though, mostly typical problems you would expect with machine generated code. Namely that it’s verbose and if you have a super complex protobuf data model (hundreds or thousands of fields) and want to ship a fatjar or similar bundling of dependencies you can run into some size issues.

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#48

Post by rbtying »

q3k wrote: post #2

From pb-jelly-gen: > The core of this crate is a python2 script codegen.py that is provided to the protobuf compiler, protoc as a plugin. That's... surprisingly janky. Not only Python tooling is always painful to deal with (compared to Go/Rust/...), but Python 2? And in a project that otherwise has no reason to depend on Python? :( In comparison, the Go protoc plugin is written in Go, the alternatice rust-protobuf pr…

another former contributor to pb-jelly, though no longer at Dropbox.

protoc plugins have an interesting bootstrapping problem as well: the protoc-gen-$LANG interface requires the ability to ser/de protobuf messages that describe the proto file's AST. If your build system builds almost everything from scratch, including the protoc plugin, this means that you need to have a variant of your protoc plugin linked to a working proto implementation...

That's not to say this is impossible or even difficult, but at the time that I last looked at it (more than a year ago at this point), it made it fairly unpalatable to move the codegen from Python to Rust.

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#49

Post by eximius »

haberman wrote: post #22

I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…

https://docs.rs/typed-arena/2.0.1/typed_arena/

Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox

#50

Post by zelly »

haberman wrote: post #22

I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…

R*st was designed to fragment the heap as much as possible. There's no way to fix it without major changes to the compiler or using "unsafe".
Post reply on HN