Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuf-py: Protobuf for Python, without compromises

buf.build

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Protobuf-py: Protobuf for Python, without compromises

#41

Post by crabbone »

squirrellous wrote: post #9
crabbone wrote: post #8

Earlier quoted context omitted.

Hey. I wrote another Python implementation of Protobuf. (protopy https://gitlab.com/doodles-archive/protopy it was a while ago and haven't touched it since). I'm not saying it's better than whatever this is or that it's any good, I just post it as a proof of sorts that I'm familiar with the problem. So, without further ado: Protobuf isn't a standard. You can't have a non-standard implementation of something that does…

Honest question - why isn’t the following document [1] a standard? Is it too loosely specified? [1] https://protobuf.dev/programming-guides/encoding/

It's like in that meme: it's not enough to say it, you need to declare it.

Or, in other words, what makes a standard a standard is that you declare that it is a standard. This declaration carries with it an obligation to respect the standard for ever and ever (like C89), regardless of whether in retrospect you realize you've made mistakes and want to fix them, or maybe wanting to add more stuff etc.

Having a standard is restrictive and uncomfortable for the designers, that's why many opt not to have a standard. Eg. Rust language doesn't have a standard (even though you may, of course, find documents detailing how it works), same for Python, Java and many other popular languages.

Re: Protobuf-py: Protobuf for Python, without compromises

#42

Post by jtbaker »

Buf.build, never again. Had a proto evangelist get us ingratiated into their system a few years ago and then they introduced a bunch of rent seeking behavior just to be able to to the code generation, and getting it ripped out was a huge PITA. https://github.com/betterproto/python-betterproto2 was a decent (async, type hinted) python implementation last I checked.

Re: Protobuf-py: Protobuf for Python, without compromises

#43

Post by joshuamorton »

crabbone wrote: post #40
usrnm wrote: post #11

Earlier quoted context omitted.

At least, Google has the resources and the will to support all their implementations in the long run. I don't always agree with what they do there, but at least I can be sure that it will still work ten years from now. At some point it becomes more important than implementation details

I don't know where you get the confidence... When it comes to Protobuf, we are now at version 3 of the format. It's been around for a while, but I'm old enough to have implemented v2 parser myself... v2 is partially supported in v3, even though the support isn't documented. But, emphasis on partially . Some things are no longer there. So... I'd say that your faith is unfounded. And, in general, there's no reason to b…

It should be noted that

1. Proto2 isn't actually deprecated or anything and is still widely used and supported

2. https://protobuf.dev/editions/overview/ replace and improve on the versioning concept and basically entirely remove the issue of versioning since versions and features can be incrementally enabled on a per-file and per-field basis.

aaand I guess 3: protobuf is absolutely critical to Google's profit.

Re: Protobuf-py: Protobuf for Python, without compromises

#44

Post by esrauch »

quietbritishjim wrote: post #33
esrauch wrote: post #23

Engineer who works on Google Protobuf here, commenting as myself and not as an official statement. It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable wit…

> ... that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++ This is a valid explanation for the odd implementation of the Python protobuf library (the only convincing one I've heard). But, the last time I looked into it (just a couple of years ago), it didn't seem reasonably possible to make use of this. I can't remember the exact issue I thi…

Well, there's still two implementations even without getting into the quoted case. But yeah, narrowly about the C++Proto shared memory thing, unfortunately it's so hard to make this work for many different reasons that we don't even advertise support for this functionality at all. This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we recommend you use C++Proto in general.

But its kind of a clear example of a surprisingly complicated technical space. The big picture is that it's actually not weird when a simpler thing which has less constraints can be better on some axes users when compared to a complicated thing that delivers on many advanced constraints.

If "readability of the .py gencode instead of the .pyi gencode" was the worst pain point (or pet peeve) here then I actually suspect Buf wouldn't have even bothered, it's just one of these that is easy to see and explain.

Re: Protobuf-py: Protobuf for Python, without compromises

#45

Post by krullin »

inexperienced protobuf user here -- why do non-googlers generally use protobuf? is it for the wire format, or for the schema language, or for the codegen?

i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most crud-like applications in the wild without requiring much fuss. even just supporting only JSON would be acceptable -- maybe an optional buyin for msgpack.

i'm thinking twirp, but abandoning protobuf and using a new, simpler schema format.

Re: Protobuf-py: Protobuf for Python, without compromises

#46

Post by masklinn »

functional_dev wrote: post #34
masklinn wrote: post #17

Earlier quoted context omitted.

> I don’t know what black magic Google uses for the Python implementation, but the classes it generates are totally opaque and impossible to inspect. TFA seems to say that they’re just thin proxies over the underlying C++ APIs, which would more than do it, and does not surprise me (the re2 Python bindings are similar, not as bad since they don’t generate Python code but they’re really c++-y — in Google’s flavour too…

I did not know this before... Google protobuf is not one thing, it is three! Same import, but: * old C++ extension * upb * pure Python upb parses FAST, but then every access is still C->Python and it slows it down. So for many reads the slow python one can win? This one helped me to dig deeper - https://vectree.io/c/how-python-protobuf-runtimes-work-pure-...

> upb parses FAST, but then every access is still C->Python and it slows it down. So for many reads the slow python one can win?

That is pretty unlikely. TFA's version is in Rust and just barely edges out upb.

Re: Protobuf-py: Protobuf for Python, without compromises

#47

Post by seanhunter »

krullin wrote: post #45

inexperienced protobuf user here -- why do non-googlers generally use protobuf? is it for the wire format, or for the schema language, or for the codegen? i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most cru…

People who want protobuf but better often settle for something like cap’n proto https://capnproto.org/

Re: Protobuf-py: Protobuf for Python, without compromises

#48

Post by jdpedrie »

jtbaker wrote: post #42

Buf.build, never again. Had a proto evangelist get us ingratiated into their system a few years ago and then they introduced a bunch of rent seeking behavior just to be able to to the code generation, and getting it ripped out was a huge PITA. https://github.com/betterproto/python-betterproto2 was a decent (async, type hinted) python implementation last I checked.

You can use their CLI and registry for free. I agree that their paid offerings are crazy expensive, but it's simple to just commit your protos in your monorepo (or a proto repo), install buf, create buf.gen.yaml with a few language plugins, and you're up and running. All the pain of protoc is handled. It's been a massive improvement in usability since they came along.

Re: Protobuf-py: Protobuf for Python, without compromises

#49

Post by esrauch »

Intralexical wrote: post #18
usrnm wrote: post #7

Earlier quoted context omitted.

The golang implementation of protobuf sucked historically (still does, but is improving) and gogo was an alternative that fixed a lot of problems and was nicer overall. Until its creator burned out and deprecated it. Chasing a constantly moving target that you have no control over is very taxing in the long run.

This is concerning to hear. What do Protobufs accomplish, that requires them to be a constantly moving target?

GoGo was not a completely separate implementation but deeply hooked into the official GoProto implementation. So it wasn't "Protobuf the binary wire format" or "Protobuf the schema language" which changed over time here, changes to the Google's Go library caused it problems. It's like building a library that integrates with Jackson (a JSON library) and Jackson details changed in ways that added toil, versus JSON changing.

Re: Protobuf-py: Protobuf for Python, without compromises

#50

Post by BobbyTables2 »

krullin wrote: post #45

inexperienced protobuf user here -- why do non-googlers generally use protobuf? is it for the wire format, or for the schema language, or for the codegen? i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most cru…

Cargo-cult mentality.

If protobuf hasn’t been created by Google, nobody would use it.

It’s actually not well designed for small requests, and doesn’t even deal with versioning all that well. It’s more verbose than a C struct yet not self-describing either.

To me, it fails to excel at any one thing yet people blindly use it anyway.

If one really considers the details, Protobuf makes ASN.1 actually start to look good!

Post reply on HN