Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuf-py: Protobuf for Python, without compromises

buf.build

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Protobuf-py: Protobuf for Python, without compromises

#51

Post by zellyn »

krullin wrote: post #45

inexperienced protobuf user here -- why do non-googlers generally use protobuf? is it for the wire format, or for the schema language, or for the codegen? i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most cru…

In my experience, during the phase where you scale up and move from interpreted to compiled code with microservices, someone inevitably benchmarks protobuf vs JSON, and protos come out ahead.

Also, using it as a schema doesn’t totally suck. There are others but proto is… fine.

I have lots of opinions, but don’t feel like spinning up to a full rant here :-)

Re: Protobuf-py: Protobuf for Python, without compromises

#52

Post by bigcityslider »

esrauch wrote: post #23

Engineer who works on Google Protobuf here, commenting as myself and not as an official statement. It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable wit…

The regular proto-to-python feels fine to me. I quit Google a while ago and have still been using protobufs, and the main things that usually make teammates wary are: 1. No way to load .proto specs at runtime (there are GH issues about this) 2. No clear and easy way to use it with HTTP/1.1. Neither is specific to Python.

Both of these might sound silly if you're used to protobufs, cause you can build little helpers for both, but plenty of people have never used protos. They would shrug proto away if there weren't someone like me to attest that it won't get in the way. It seems like Google focused on gRPC, but the real prize for adoption would've been going after the simple HTTP+JSON use cases. Like an official protobuf Express middleware.

Re: Protobuf-py: Protobuf for Python, without compromises

#53

Post by Chu4eeno »

masklinn wrote: post #46
functional_dev wrote: post #34

Earlier quoted context omitted.

I did not know this before... Google protobuf is not one thing, it is three! Same import, but: * old C++ extension * upb * pure Python upb parses FAST, but then every access is still C->Python and it slows it down. So for many reads the slow python one can win? This one helped me to dig deeper - https://vectree.io/c/how-python-protobuf-runtimes-work-pure-...

> upb parses FAST, but then every access is still C->Python and it slows it down. So for many reads the slow python one can win? That is pretty unlikely. TFA's version is in Rust and just barely edges out upb.

rust code tends to be incredibly slow unless you spend a ton of time optimizing (compared to other compiled languages, and contrary to popular belief).

Re: Protobuf-py: Protobuf for Python, without compromises

#54

Post by Chu4eeno »

seanhunter wrote: post #47
krullin wrote: post #45

inexperienced protobuf user here -- why do non-googlers generally use protobuf? is it for the wire format, or for the schema language, or for the codegen? i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most cru…

People who want protobuf but better often settle for something like cap’n proto https://capnproto.org/

didn't the capnproto author end up creating protobuf2? or am I mixing things up.

Re: Protobuf-py: Protobuf for Python, without compromises

#55

Post by quietbritishjim »

esrauch wrote: post #44
quietbritishjim wrote: post #33

Earlier quoted context omitted.

> ... that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++ This is a valid explanation for the odd implementation of the Python protobuf library (the only convincing one I've heard). But, the last time I looked into it (just a couple of years ago), it didn't seem reasonably possible to make use of this. I can't remember the exact issue I thi…

Well, there's still two implementations even without getting into the quoted case. But yeah, narrowly about the C++Proto shared memory thing, unfortunately it's so hard to make this work for many different reasons that we don't even advertise support for this functionality at all. This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we re…

> This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we recommend you use C++Proto in general.

Statically linking everything doesn't work in this case because each C extension module (i.e. each Python package with some native code in it) is, in effect, a shared library.

If you statically linked the C++ protobuf library into both the Python protobuf library and the Python tensorflow library then you'd end up with two copies of the library in the process (with two copies of globals/statics such as the protobuf global descriptor pool). That's not a problem in itself, but it wouldn't be possible to pass protobuf objects between them.

(In principle, you could perhaps statically link protobuf into the CPython executable, but I really don't think Google is doing that.)

The usual solution is the exact opposite to static linking: you have to build the C++ protobuf library as a shared library (libprotobuf.so) and then link to it from both of your C extension modules (i.e. from the other two shared libraries).

Following the tiny crumbs of evidence of how to do this for protobuf, it seems like that was previously how you would do it. But it's possibly not now? It seems like proto_api.h [1] has a way to get the global descriptor pool, possibly to work around needing a common shared library? But also now that header is deprecated anyway [2]? (I think this is roughly where I got to before when I gave up.)

[1] https://github.com/protocolbuffers/protobuf/blob/main/python...

[2] https://github.com/protocolbuffers/protobuf/issues/9464

Re: Protobuf-py: Protobuf for Python, without compromises

#56

Post by seanhunter »

Chu4eeno wrote: post #54
seanhunter wrote: post #47

Earlier quoted context omitted.

People who want protobuf but better often settle for something like cap’n proto https://capnproto.org/

didn't the capnproto author end up creating protobuf2? or am I mixing things up.

Hence the “something like”. There are a few of these sorts of things. I’m not really in the market for them so I don’t try to keep up.

Re: Protobuf-py: Protobuf for Python, without compromises

#57

Post by igetspam »

usrnm wrote: post #4

After gogoproto I'm hesitant to depend on another non-standard implementation, getting off gogo was a pain. This thing may be better than the one from Google (gogo definitely was), but can we be sure that it will still be around in 10 years?

Can you be sure Google’s ${anything} will be around in 6 months? They have a decades long habit of dumping things that have major use but aren’t novel internally. Worrying about the next 10 years isn’t something you can honestly do when you your argument includes Google owning a thing.

Bias: I fought for things like Reader from the inside. If it doesn’t move needles, it goes away.

Re: Protobuf-py: Protobuf for Python, without compromises

#58

Post by kentonv »

Chu4eeno wrote: post #54
seanhunter wrote: post #47

Earlier quoted context omitted.

People who want protobuf but better often settle for something like cap’n proto https://capnproto.org/

didn't the capnproto author end up creating protobuf2? or am I mixing things up.

That's me. Other way around -- proto2 came first.

Re: Protobuf-py: Protobuf for Python, without compromises

#59

Post by bufbuild »

bigcityslider wrote: post #52
esrauch wrote: post #23

Engineer who works on Google Protobuf here, commenting as myself and not as an official statement. It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable wit…

The regular proto-to-python feels fine to me. I quit Google a while ago and have still been using protobufs, and the main things that usually make teammates wary are: 1. No way to load .proto specs at runtime (there are GH issues about this) 2. No clear and easy way to use it with HTTP/1.1. Neither is specific to Python. Both of these might sound silly if you're used to protobufs, cause you can build little helpers f…

> No clear and easy way to use it with HTTP/1.1. Neither is specific to Python.

> It seems like Google focused on gRPC, but the real prize for adoption would've been going after the simple HTTP+JSON use cases. Like an official protobuf Express middleware.

We'd flag https://connectrpc.com for you :-) HTTP/1.1 out of the box, switches on Content-Type between binary and JSON, and fully-compatible with gRPC. cURL with JSON just works.

Re: Protobuf-py: Protobuf for Python, without compromises

#60

Post by parhamn »

Whats the case for protobuf these days? I loved it for a while. Don't dislike it particularly now (but I've rolled off).

- Performance? Fastest JSON lib in most languages are as fast if not faster

- Cross language schema generation? So many tools for this these days, they do one thing do them right (depending on your choice of 'right' for things like unions/enums/etc)

- The wire protocol? Seems to get in the way vs http2/3. Need special considerations for your proxies (be it nginx or cloudflare). Forced certs, etc are annoying too.

I feel like to most shops these days its mostly a schema manager? Protobuf is super bloated for that use case.

Post reply on HN