This is great. AFAIK this is the only protobuf library in Rust that supports zero copy. Maybe this'll help some of the other libraries implement similar features?
rust-protobuf supports zero-copy when using `bytes` feature and reading from `Bytes`.
Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
31–40 of 54 posts
Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
#32Earlier quoted context omitted.
Quick-protobuf has been around for a while and supports it.
Thanks, I hadn't seen this. I'm curious about how they compare - it looks like quick-protobuf uses Cow, but I think pb-jelly doesn't?
Jelly uses an external crate called Bytes to achieve it. It has nicer usability compared to Cow I guess.
Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
#33Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
#34I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…
Placement new is just an optimization to avoid the initial memcpy from the stack in cases in which LLVM can't work it out itself. I don't believe that placement new ever enables semantics that aren't possible with plain old move semantics.
Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
#35I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…
Rust objects can always be moved to arbitrary places in memory, so I'm not sure how "in place" construction of a Rust object can even make sense unless it explicitly involves the Pin
feature to prevent moves. This is quite different from C/C++ where there's no default expectation that a constructed object can be bitwise-copied somewhere else in memory. Even Box is really a special case since the object is meant to be accessed via an owning pointer or a (shared or mutable) reference; when accessing the underlying T by value, that always involves a move of the object so again there's no such thing as being "in place".
Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
#36I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…
Rust objects can always be moved to arbitrary places in memory, so I'm not sure how "in place" construction of a Rust object can even make sense unless it explicitly involves the Pin feature to prevent moves. This is quite different from C/C++ where there's no default expectation that a constructed object can be bitwise-copied somewhere else in memory. Even Box is really a special case since the object is meant to be…
Placement new is just avoiding a copy.
Without placement new allocating an object involves:
a) Constructing the object on the stack
b) Copying it to the heap
With placement new you construct the object directly on the heap, thus avoiding a copy.
It's purely an optimization to avoid a single copy. It's a bit surprising that Rust doesn't have it, given the focus on control over such things, and that it had an experimental 'box' keyword for this at 1.0.
edit: To be clear, the extra copy is not guaranteed - compilers can elide it. In order to help the compiler do this you can look at crates like boxext, which provide extension methods on Box that help the compiler to remove the copy. But placement new guarantees this.
Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
#37From pb-jelly-gen: > The core of this crate is a python2 script codegen.py that is provided to the protobuf compiler, protoc as a plugin. That's... surprisingly janky. Not only Python tooling is always painful to deal with (compared to Go/Rust/...), but Python 2? And in a project that otherwise has no reason to depend on Python? :( In comparison, the Go protoc plugin is written in Go, the alternatice rust-protobuf pr…
Hi! One of the authors here.
This was an oversight in the documentation. The codegen is py2 and py3 compatible. Fixed!
See issues https://github.com/dropbox/pb-jelly/issues/37 and https://github.com/dropbox/pb-jelly/issues/40 for context.
Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
#38I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…
Placement new is just an optimization to avoid the initial memcpy from the stack in cases in which LLVM can't work it out itself. I don't believe that placement new ever enables semantics that aren't possible with plain old move semantics.
Ah, in that case it sounds like heterogenous arenas are more or less a solved problem in Rust, even if they aren't necessarily 100% optimal.
Probably the more difficult piece then is just how to model arena ownership of a tree of objects that all have links between them. We want to guarantee that links to sub-messages remain valid, which we would expect to be true if they are all in the same arena. But I believe Rust allows moving/swapping objects in and out of the arena?
Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
#39I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…
Hey @haberman! One of the authors here.
We actually do have an arena-esque implementation built on top of pb-jelly internally, as it was needed for Magic Pocket.
It's built on top of the Blob traits exposed by pb-jelly. It's not yet open-source, but it would be a good candidate to do next! It also definitely has unsafe code to your point. We open sourced the safe implementations that uses more standard types (Bytes/Buffer/Vec) first.
There's a decent amount of cleanup needed before we can opensource that as well, as much of it was built years ago, when rust ecosystem was less mature (eg Bytes/Buffer weren't around yet).
I like where you're thinking!
Re: Pb-jelly – A Protobuf code generation framework for Rust, developed at Dropbox
#40I work on the protobuf team at Google, and I'm a big fan of Rust, though I haven't written much actual Rust except a bunch of Project Euler solutions. For protobuf in C++, we've been moving more and more in the direction of using arenas for memory allocation. When you parse a protobuf, it creates a tree of objects that are usually all deleted at the same time. Freeing an arena is much, much cheaper than traversing th…
Hey @haberman! One of the authors here. We actually do have an arena-esque implementation built on top of pb-jelly internally, as it was needed for Magic Pocket. It's built on top of the Blob traits exposed by pb-jelly. It's not yet open-source, but it would be a good candidate to do next! It also definitely has unsafe code to your point. We open sourced the safe implementations that uses more standard types (Bytes/B…
That's great, I'll look forward to seeing the arena-oriented code someday. :)