Live data from Hacker News

Using Protobuf instead of JSON to communicate with a front end

blog.wearewizards.io

91–99 of 99 posts

Re: Using Protobuf instead of JSON to communicate with a front end

#91
There is definitely a place for binary serialization/de-serialization and transmission. Inter-system communication is probably the best place for binary or any place that needs high speed real-time communication with the smallest size to fit in MTU limits (game protocols over UDP for instance). Any place that you control the client and server is ok to use binary.

However, I do feel there is a strange swaying back to binary (Protobuf/HTTP/2/etc). Developers are trying to wedge it in now in places it may cause more problems because it is more efficient in performance but not in use or implementation. Plus, like mentioned in this thread, you can compress JSON to be very small to send over the wire which makes the compactness of it a non-issue in non real-time cases. Going binary just to go binary is more trouble than it is worth in most cases.

- Binary over keyed plain text (JSON) is harder to generically parse objects i.e. dictionaries/lists for just a few fields/keys.

- Binary over JSON also seems to lock down messaging more, people have more work to change binary explicit messages because of offset issues and client/server tools must be in sync rather than just adding a new key that can be pulled as needed.

- Third party implementation and parsing of JSON/XML is more forgiving making version upgrades and changes easier to do. This is especially apparent on projects that are taken over by other developers.

- The language/platform on the backend leaks into the messaging. For instance Protobuf only runs on js/python currently and has various versions. The best messaging is independent of the platform and versioning is easier.

I would bet binary formats end up causing more bugs over keyed/plaintext (JSON/XML and possibly compressed) though I have nothing to back that up by except my own experience largely in game development where networking state is almost always binary, for server/data I wouldn't use it unless it needs to be real-time.

That being said Protobuf is awesome and I hope developers are using it where it is best suited and that developers don't start obfuscating messaging for performance where it doesn't really need to be, better to be simple unless you need to make it more complex at every level.

Re: Using Protobuf instead of JSON to communicate with a front end

#92
With one end in Python 2 and the other end in Javascript, using binary protobufs seems misplaced optimization. It's nice to know the support is there (well, not in Python 3, apparently), in case you need to talk to something that speaks protobufs.

I'm looking forward to seeing protobufs in Rust as a macro. It should be possible; there's an entire regular expression compiler for Rust as a compile-time macro, which is a useful optimization.

Re: Using Protobuf instead of JSON to communicate with a front end

#93
post #86

Having used both on a few projects, including a JS frontend, my advice is: "Don't use protobufs if you don't have to". Protobufs can be much faster, and provide a strict schema, but it comes at the price of higher maintenance costs. JSON is much simpler, easier to implement, and MUCH easier to debug. If your GPB looks like it's building properly, but fails to parse, it's a huge pain to try and decode/debug the binary…

In my experience, it's not that tough to write a 'proto-to-dict' function in python, which lets you crack open the proto and look at its juicy innards...

Re: Using Protobuf instead of JSON to communicate with a front end

#94
post #92

With one end in Python 2 and the other end in Javascript, using binary protobufs seems misplaced optimization. It's nice to know the support is there (well, not in Python 3, apparently), in case you need to talk to something that speaks protobufs. I'm looking forward to seeing protobufs in Rust as a macro. It should be possible; there's an entire regular expression compiler for Rust as a compile-time macro, which is…

It's not protobuf, but Rust has quite good Cap'n Proto support: https://crates.io/crates/capnp

Re: Using Protobuf instead of JSON to communicate with a front end

#95
post #52

Making JSON first-class is an explicit design goal of proto3, the next version of Protocol Buffers currently in alpha: https://developers.google.com/protocol-buffers/docs/proto3#j... This will allow you to switch between JSON and protobuf binary on the wire easily, while using official protobuf client libraries. So you can choose easily whether you care more about size/speed efficiency or wire readability. Best of bo…

> Message field names are mapped to lowerCamelCase Why is a mapping to camel case necessary? I imagine it creates the potential for collisions, no?

Co-author of proto3 here. The reasons we chose lowerCamelCase are compatibility with Google's REST APIs (like Gmail API) and readability for users who work with JSON output directly without client library. API designers should not define confusing data schemas, let alone allow collisions. Most proto messages have small number of fields, avoiding name collision is a trivial for an API designer.

Re: Using Protobuf instead of JSON to communicate with a front end

#96
post #86

Having used both on a few projects, including a JS frontend, my advice is: "Don't use protobufs if you don't have to". Protobufs can be much faster, and provide a strict schema, but it comes at the price of higher maintenance costs. JSON is much simpler, easier to implement, and MUCH easier to debug. If your GPB looks like it's building properly, but fails to parse, it's a huge pain to try and decode/debug the binary…

Co-author of proto3 here. Proto3 was specifically designed to make proto more friendly in variety of environments, which includes native JSON support. New Google REST APIs are defined in proto3, which are open sourced[1].

[1] https://github.com/google/googleapis

Re: Using Protobuf instead of JSON to communicate with a front end

#97
post #76

Earlier quoted context omitted.

TextDecoder/TextEncoder is the emerging standard way to do this. https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder

Interesting, how does that relate to this? https://encoding.spec.whatwg.org/

That's the same thing.

Re: Using Protobuf instead of JSON to communicate with a front end

#98

Making JSON first-class is an explicit design goal of proto3, the next version of Protocol Buffers currently in alpha: https://developers.google.com/protocol-buffers/docs/proto3#j... This will allow you to switch between JSON and protobuf binary on the wire easily, while using official protobuf client libraries. So you can choose easily whether you care more about size/speed efficiency or wire readability. Best of bo…

When can we expect proto3 to be stable and ready to use for general public?

Re: Using Protobuf instead of JSON to communicate with a front end

#99

One of the comments on that article was "YAY! JSON is wastefully large. I'd love to replace it." Is this true? I'm confused why JSON would be seen as a wasteful as a format. It seems to be that with any decent compression I would think it's hard to get much smaller. In this case I'm not talking about the other advantages Protobuf offers, I just want to know about size.

There are basically 2 areas where JSON is really wasteful. Compression can help with both of those. 1. Dictionary keys are repeated when you have an array of similar objects. 2. Non-text data. JSON can't natively represent binary data, forcing people to use things like base64 for binary and base10 for numbers.

I hadn't considered binary data. Thanks.
Post reply on HN