Pyrobuf: A Cython Alternative to Google's Python Protobuf Library

It's really interesting to me that the top two comments on this story both wish Protocol Buffers were different, but seem to be asking for opposite things.

Your comment wishes that there was a pure-Python implementation that doesn't do metaprogramming. That would imply doing code generation, but having the generated code be more "concrete", so-to-speak (containing all of the actual implementation).

However, a different comment (https://news.ycombinator.com/item?id=10762101) admires the thriftpy project which doesn't have a codegen step at all. This is in some sense the opposite of what you want: everything becomes metaprogramming.

I work on Protocol Buffers at Google, and I frequently observe these kinds of opposing feelings about how protobufs ought to work. One thing I've learned is how incredibly difficult it is to please everybody. Another good example of this is whether to be pure-Python: some people (like you) want that. But it's nearly impossible to make it very fast, and lots of users are sensitive to speed (you can find various comments and blog articles complaining about the speed of Python protobuf).

There are a couple of reasons I chose FB over Can'n Proto, Python and full C++ doesn't work on MSVC/Windows (this might have changed recently especially given the Clang compiler integration into MS backend)

Second problem is Capnproto doesn't have structs, ie. you want something like struct Vec3 {x: float, y: float, z: float} in CapnProto you have to either make Vec3 a ref type (+ pointer size and lookup indirection for each instance) or inline it manually (especially annoying given that you must manually number CapnProto fields, maybe this is a benefit in some use cases but it's of no use to me and only creates work).

CapnProto is much bigger as well - it includes it's own RPC/distributed object protocol and library - the serialization part isn't separate they are in one library - that's taking in a lot of useless code that you would have to figure out if you decided to maintain it.

Also worth noting FlatBuffers on Python are actually not that fast because they are pure-python - there are Cython versions in the PR (at least last time I checked) - this doesn't matter in my use case but worth pointing out. And the Python API is very very unpythonic (C style API with even the naming completely off from PEP8) compared to CapnProto which is excellent !

I needed a cross language typed serialization format for binary files and FlatBuffers is better for that than CapnProto IMO.

Hey Josh! Jorge from gRPC here. What's your take on Pyrobuf?

I suspect it comes down to the details of the serialization format.

I know that Thrift and Protobuf were developed / publicly released around the same time (~2008). They both have serialization and RPC approaches. They both have an IDL format and a compiler for codegen in static languages like C++ and Java.

Thrift was adopted by the Apache O/SS community and so it can be found in some Apache projects, e.g. Thrift is used as an interop format in Apache Storm and as a serialization backend supported by Apache Parquet.

Thrift has had a lot of client libraries for different languages come out over the years, so it tends to work everywhere.

IIRC, CapNProto and FlatBuffers were each built as fresh takes on protobuf. So I imagine they are similar, with IDLs and serialization approaches of their own. I took a quick look at CapNProto's Python module documentation and it looks very similar to thriftpy in philosophy, but I don't know its intricate details as I haven't used it.

This guide to Thrift covers all the important details of its schema approach and is meant as a cross-language guide. It should be the official docs, but alas, it's not!

https://diwakergupta.github.io/thrift-missing-guide/

> the serialization part isn't separate they are in one library

Although Cap'n Proto's C++ implementation is hosted in a single git repository, it does compile to several distinct libraries. You can use just the core serialization/deserialization part, libcapnp, if that's all you need. There are separate components, libcapnpc and libcapnp-rpc, for dynamic reflection and the object-capability remote procedure call system.