Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuf-py: Protobuf for Python, without compromises

buf.build

Post reply on HN
Print view View original

61–65 of 65 posts

Re: Protobuf-py: Protobuf for Python, without compromises

#61

Post by bigcityslider »

bufbuild wrote: post #59
bigcityslider wrote: post #52

Earlier quoted context omitted.

The regular proto-to-python feels fine to me. I quit Google a while ago and have still been using protobufs, and the main things that usually make teammates wary are: 1. No way to load .proto specs at runtime (there are GH issues about this) 2. No clear and easy way to use it with HTTP/1.1. Neither is specific to Python. Both of these might sound silly if you're used to protobufs, cause you can build little helpers f…

> No clear and easy way to use it with HTTP/1.1. Neither is specific to Python. > It seems like Google focused on gRPC, but the real prize for adoption would've been going after the simple HTTP+JSON use cases. Like an official protobuf Express middleware. We'd flag https://connectrpc.com for you :-) HTTP/1.1 out of the box, switches on Content-Type between binary and JSON, and fully-compatible with gRPC. cURL with JS…

Yeah I saw that a while back, this is how it should be done

Re: Protobuf-py: Protobuf for Python, without compromises

#62

Post by Scaevolus »

jdpedrie wrote: post #48
jtbaker wrote: post #42

Buf.build, never again. Had a proto evangelist get us ingratiated into their system a few years ago and then they introduced a bunch of rent seeking behavior just to be able to to the code generation, and getting it ripped out was a huge PITA. https://github.com/betterproto/python-betterproto2 was a decent (async, type hinted) python implementation last I checked.

You can use their CLI and registry for free. I agree that their paid offerings are crazy expensive, but it's simple to just commit your protos in your monorepo (or a proto repo), install buf, create buf.gen.yaml with a few language plugins, and you're up and running. All the pain of protoc is handled. It's been a massive improvement in usability since they came along.

if you happen to regenerate your protos 5 times in an hour you'll get ratelimited

it's a very frustrating system. you should spend a modicum of effort figuring out how to generate your things locally yourself instead, be that docker, arcane wasm...

Re: Protobuf-py: Protobuf for Python, without compromises

#63

Post by crabbone »

joshuamorton wrote: post #43
crabbone wrote: post #40

Earlier quoted context omitted.

I don't know where you get the confidence... When it comes to Protobuf, we are now at version 3 of the format. It's been around for a while, but I'm old enough to have implemented v2 parser myself... v2 is partially supported in v3, even though the support isn't documented. But, emphasis on partially . Some things are no longer there. So... I'd say that your faith is unfounded. And, in general, there's no reason to b…

It should be noted that 1. Proto2 isn't actually deprecated or anything and is still widely used and supported 2. https://protobuf.dev/editions/overview/ replace and improve on the versioning concept and basically entirely remove the issue of versioning since versions and features can be incrementally enabled on a per-file and per-field basis. aaand I guess 3: protobuf is absolutely critical to Google's profit.

It might be uncalled for, but let me use a philosophical reference: you are trying to derive ought from is. Many tried to prove it to be possible, but most face such propositions with skepticism.

Let me give you some examples of things that were absolutely detrimental to the business of a large company that vanished in a matter of years if not months:

* Adobe Flash. Flash players was for a few years the most installed program on Earth. It vanished without a trace, dropped like a hot potato by everyone who swore they will use it for ever and ever.

* ASP Classic, especially in combination with VBScript. It was the hottest thing twenty (or am I getting too old?) years ago. Everyone was selling books and courses on ASP Classic, probably half the Internet was written in it... and it's gone.

* I expect x86 ISA to eventually die and be completely replaced by ARM or maybe something else. Some big players are already jumping off the bandwagon, and it looks like things are only going to get worse.

There were, of course, more, but I think these three examples should suffice. If tomorrow Google comes up with a better format, or, for whatever reason, tanks its business, or decides to switch to vertical farming in Arizona... say goodby to Protobuf. They owe you zilch. They never promised you anything. You were allowed to use their tech because it was convenient to them, but once it's not, you will be holding the bag.

This is the difference between a standard and a spec published by the designer. The designer is free to make any changes they see fit, up to and including completely destroying their work. They can pull the run from under you making the users pay for the use, they can make modifications to their tools that will require from you to buy things you wouldn't normally want to buy. The possibilities are endless.

Re: Protobuf-py: Protobuf for Python, without compromises

#64

Post by squirrellous »

krullin wrote: post #45

inexperienced protobuf user here -- why do non-googlers generally use protobuf? is it for the wire format, or for the schema language, or for the codegen? i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most cru…

These days I am of the opinion that we don’t need a real IDL. Instead one should choose a “primary language” and define the schema in that language. All other language bindings are generated from that canonical language. Then for wire format you’d just pick something workable from pb json msgpack etc. This replaces all the Google codegen intermediate layer, and for the primary language it naturally becomes very ergonomic.

Re: Protobuf-py: Protobuf for Python, without compromises

#65

Post by esrauch »

quietbritishjim wrote: post #55
esrauch wrote: post #44

Earlier quoted context omitted.

Well, there's still two implementations even without getting into the quoted case. But yeah, narrowly about the C++Proto shared memory thing, unfortunately it's so hard to make this work for many different reasons that we don't even advertise support for this functionality at all. This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we re…

> This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we recommend you use C++Proto in general. Statically linking everything doesn't work in this case because each C extension module (i.e. each Python package with some native code in it) is, in effect, a shared library. If you statically linked the C++ protobuf library into both the Pyt…

> principle, you could perhaps statically link protobuf into the CPython executable, but I really don't think Google is doing that.)

Yes, we actually do exactly this (not just protobuf but generally all of the other C code with it for the given application). But there are other approaches that can work.

> But also now that header is deprecated anyway

It's deprecated exactly because the preferred/supported/default runtime is the upb based runtime. The odd gencode design is what made that transition possible as an implementation detail. So the deprecation is just the same topic here, that people successfully setting up the dynamic linking is not really realistic/viable unfortunately, and the effort for open source support is spent making the best behavior on the upb runtime which can't have that feature.

Post reply on HN