Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The reason to use Protobuf is not performance

buf.build

Post reply on HN
Print view View original

11–20 of 23 posts

Re: The reason to use Protobuf is not performance

#11

Post by arp242 »

stubish wrote: post #3
theamk wrote: post #2

schema-driven development is great, but I wish people spent more time on JSON-based API schema rather than switch to protobuf. Because protobufs immediately require some minimal level of complexity (at least a build system!) and thus effectively block small one-off scripts. And the debuggability is also not great. Can we (as an industry) pick a standard (openapi? jsonschema? something else?) and add the tooling to ma…

I don't think a large group can pick a standard. What works well for one programming language does not work for another, and we devolve into tribes. Protobuf gained popularity when it dropped features to better support Go more naturally (no more null/nil for example), and rode in on a wave of Go adoption. And made things worse from a Python developer perspective. Pretty much the same as toml adoption, which fits more…

TOML doesn't "fit static languages" better and is just as dynamically typed as YAML. It's used extensively in the Python ecosystem, which has TOML in stdlib (not YAML).

I'm less sure about protobufs in Python (I never tried), but in general relying too much on Python's dynamic nature is not a good thing IMHO.

Re: The reason to use Protobuf is not performance

#13

Post by closeparen »

theamk wrote: post #2

schema-driven development is great, but I wish people spent more time on JSON-based API schema rather than switch to protobuf. Because protobufs immediately require some minimal level of complexity (at least a build system!) and thus effectively block small one-off scripts. And the debuggability is also not great. Can we (as an industry) pick a standard (openapi? jsonschema? something else?) and add the tooling to ma…

Protobuf has a JSON canonicalization [0]. All our gRPC servers automatically accept HTTP/JSON traffic with the procedure name specified in a header. That's what I use when I'm writing a script.

https://protobuf.dev/programming-guides/proto3/#json

Re: The reason to use Protobuf is not performance

#14

Post by dontlaugh »

closeparen wrote: post #13
theamk wrote: post #2

schema-driven development is great, but I wish people spent more time on JSON-based API schema rather than switch to protobuf. Because protobufs immediately require some minimal level of complexity (at least a build system!) and thus effectively block small one-off scripts. And the debuggability is also not great. Can we (as an industry) pick a standard (openapi? jsonschema? something else?) and add the tooling to ma…

Protobuf has a JSON canonicalization [0]. All our gRPC servers automatically accept HTTP/JSON traffic with the procedure name specified in a header. That's what I use when I'm writing a script. https://protobuf.dev/programming-guides/proto3/#json

Twirp has this built-in https://twitchtv.github.io/twirp/docs/curl.html

It's very handy in some situations.

Re: The reason to use Protobuf is not performance

#15

Post by quectophoton »

theamk wrote: post #2

schema-driven development is great, but I wish people spent more time on JSON-based API schema rather than switch to protobuf. Because protobufs immediately require some minimal level of complexity (at least a build system!) and thus effectively block small one-off scripts. And the debuggability is also not great. Can we (as an industry) pick a standard (openapi? jsonschema? something else?) and add the tooling to ma…

If only JSON-RPC[1] were more popular for JSON-based APIs. With JSON-RPC specifically, the trade-off is that it would not work well with binary data; but if that's a problem then instead of JSON it could be BSON, bencode, EBML, or whatever (at that point it wouldn't be JSON-RPC though), the point is that you can nest stuff without switching syntax.

But compared to HTTP, if what you want is just JSON, then JSON-RPC saves you from having 5 different places to pass data, each with its own special syntax (one for method name; one for path; one for query params in the path; one for headers; one for body with the actual data).

We only not notice this because of just how many libraries are doing the hard work for us and making it seem like HTTP requests are clean and nice stuff; but no, HTTP is certainly not a nice protocol. Its main advantage is its ubiquity.

People may ask, "what about caching and stuff?". Well, with HTTP we made that work by passing a special key in the part of the request that is delimited by CRLF and where each line contains a key-value pair separated by a colon-and-space; and the clients know that a response was cached because they can read the first line of the response, which is a space-separated list of keywords that has a specific numeric literal in its second position (yes, I'm oversimplifying).

So I'm pretty sure we can do something better with fewer different syntaxes. And reverse proxies would presumably have it harder to be vulnerable to things like request smuggling[2].

[1]: https://en.wikipedia.org/wiki/JSON-RPC [2]: https://en.wikipedia.org/wiki/HTTP_request_smuggling

Re: The reason to use Protobuf is not performance

#16

Post by arein3 »

closeparen wrote: post #13
theamk wrote: post #2

schema-driven development is great, but I wish people spent more time on JSON-based API schema rather than switch to protobuf. Because protobufs immediately require some minimal level of complexity (at least a build system!) and thus effectively block small one-off scripts. And the debuggability is also not great. Can we (as an industry) pick a standard (openapi? jsonschema? something else?) and add the tooling to ma…

Protobuf has a JSON canonicalization [0]. All our gRPC servers automatically accept HTTP/JSON traffic with the procedure name specified in a header. That's what I use when I'm writing a script. https://protobuf.dev/programming-guides/proto3/#json

Could you provide an example of json request to a grpc endpoint (url + headers + body)?

Can it be done via http (ex. Postman) or it needs http2?

Re: The reason to use Protobuf is not performance

#17

Post by kentonv »

Eridrus wrote: post #9
hresvelgr wrote: post #7

While certainly not as ubiquitous as Protobuf, Cap'n Proto[1] is an extremely excellent alternative worth checking out. The quality of the generated C++ code is enough reason for me to prefer it. [1] https://capnproto.org/

Does protobuf or capnproto have good tooling support out in the world? Proto was very convenient at Google since everything was proto, your debug tools were proto, all your data pipeline tools spoke proto, you had command line utilities for protos, etc, etc. It seems like the rest of the world is on JSON unless they actually need performance, so if you don't get performance, should you use something like a code gener…

> Does protobuf or capnproto have good tooling support out in the world?

Buf (author of the article) has made it their mission to build out that tooling and support for Protobuf. I think they're doing a good job.

Cap'n Proto's ecosystem is admittedly weak, since it as yet has no full-time team trying to build this out.

(Disclosure: I'm the author of Cap'n Proto and, long ago, Protobuf v2. I'm also a small investor in Buf.)

Re: The reason to use Protobuf is not performance

#19

Post by closeparen »

arein3 wrote: post #16
closeparen wrote: post #13

Earlier quoted context omitted.

Protobuf has a JSON canonicalization [0]. All our gRPC servers automatically accept HTTP/JSON traffic with the procedure name specified in a header. That's what I use when I'm writing a script. https://protobuf.dev/programming-guides/proto3/#json

Could you provide an example of json request to a grpc endpoint (url + headers + body)? Can it be done via http (ex. Postman) or it needs http2?

You can check the Twirp doc in the sibling comment. Ours is similar. Works on regular HTTP, curl or Postman is fine.

Re: The reason to use Protobuf is not performance

#20

Post by ath3nd »

Protobuf is great, but wait till you hear about JSON!

Jokes aside, while JSON is bulkier, binary protocols like protobuf, avro, thrift have one big penalty: they are hard to read/understand by humans without some tooling. The value of seeing the network call and being able to actually understand what's going on without having to translate the format into something human readable is hard to overstate.

The bigger payload size of JSON is not enough for me not to see the main benefit : I can check what's going on with any tool: from curl, to postman, to my browser's damn network window. With some compression, JSON is not that bad size wise either.

To read and generate protobuf, you mostly have to rely on some tooling/library provided by Google, and those, like most of their software (looking at you, Guava), is the reverse of backwards compatible, despite what they're advertising. Just search Google for "protobuf version mismatch".

Maybe I am speaking heresy here, but software by Google (like k8s, protobuf, guava, etc) for me has been notoriously backwards incompatible and overly complex, and I try to steer clear from it as much as possible. I gotta give the crown to K8s here, it specifically is an absolute clusterfuck, the pinnacle of what happens when 1000s of good ideas are combined together in a sloppy, quick execution and without regard for previous versions.

You can achieve schema with any other binary format like avro (which doesn't tie you to using google libraries, which i hate with a passion), and you can achieve a schema with xml and json, so after reading the article, there is still 0 reason for me to use protobuf, if not for performance and only within a google ecosystem.

Post reply on HN