Using Protobuf instead of JSON to communicate with a front end • Hacker News Board

Re: Using Protobuf instead of JSON to communicate with a front end

#51

Post by philsnow » Fri, Jun 05, 2015, 4:34 PM UTC

teh wrote: post #20

skybrian wrote: post #19

It's possible to encode a protobuf as JSON and we do it all the time at Google. In browsers, native JSON parsing is very fast and the data is compressed, so going to a binary format doesn't seem worthwhile. The .proto file is used basically as an IDL from which we generate code.

Can I ask which library you are using? I found a few [1] but none seem super robust. Also, how do you deal with the bytes type? [1] https://code.google.com/p/protobuf-json/ https://github.com/benhodgson/protobuf-to-dict

Just a guess here: I would have an agreed-upon key name suffix like "__b64_enc" or something. Serializers take a field "foo" of type bytes and serialize it as "foo__b64_enc": b64(value), and deserializers strip and base64-decode.

edit: it's exactly what you would do if you wanted to pass any binary data as json over the wire, regardless of whether you're using protobufs. you'd just get it "for free" (meaning you wouldn't have to write the boilerplate, not that you don't have to en/decode).

Re: Using Protobuf instead of JSON to communicate with a front end

#52

Post by caust1c » Fri, Jun 05, 2015, 4:45 PM UTC

haberman wrote: post #43

Making JSON first-class is an explicit design goal of proto3, the next version of Protocol Buffers currently in alpha: https://developers.google.com/protocol-buffers/docs/proto3#j... This will allow you to switch between JSON and protobuf binary on the wire easily, while using official protobuf client libraries. So you can choose easily whether you care more about size/speed efficiency or wire readability. Best of bo…

> Message field names are mapped to lowerCamelCase

Why is a mapping to camel case necessary? I imagine it creates the potential for collisions, no?

Re: Using Protobuf instead of JSON to communicate with a front end

#53

Post by w0utert » Fri, Jun 05, 2015, 4:52 PM UTC

imaginenore wrote: post #34

Have you guys tried MsgPack? If so, is it worth it? http://msgpack.org/

MessagePack worked great for us, fast, compact serialization, easy to use, great platform & language support. I've never used protocol buffers, mainly because I really dislike that you have to write .proto files that are then translated to code, which IMO In many situations is an unnecessary kludge. I understand it can be useful especially if you need to serialize the same things from different languages, and don't want to write the same serialization code twice (or more), but if that's not a concern for your project, I have no idea why I should prefer protocol buffers over MsgPack

Re: Using Protobuf instead of JSON to communicate with a front end

#54

Post by oppositelock » Fri, Jun 05, 2015, 4:54 PM UTC

I worked on a product inside Google which used protos (v1) as the data format to a web front end, and in practice, that system was a failure, in part to the decision to use protos. The deserialization cost of protocol buffers is too high if you're doing complex data throughput, and even though the data size is smaller, it's better to send larger gzipped JSON (which will be decompressed in native code) and deserialized into JS (also via native code). We weren't using ProtoBuf.js, but our own internal javascript implementation of a similar library, and doing all of this in JS was too expensive. Granted, we were sending around protos that had multi megabyte payloads at times.

We rewrote our app eventually to send protos in JSON format to the app, while just letting our backends still pass around native protos, it worked a lot better.

Re: Using Protobuf instead of JSON to communicate with a front end

#55

Post by mhahn » Fri, Jun 05, 2015, 4:59 PM UTC

I'm curious if Google has a common envelope they send all service messages with. Ie. A common way of specifying pagination parameters, auth tokens etc. when sending protobuf messages between services. I've been using protobufs for my services and wrote a ServiceRequest object which has worked well. I was more just surprised about not being able to find much documentation on actual deployments as opposed to just simple tutorials.

Re: Using Protobuf instead of JSON to communicate with a front end

#56

Post by labianchin » Fri, Jun 05, 2015, 5:00 PM UTC

I wonder how would that be like with Avro. It also has JSON encoding: https://avro.apache.org/docs/1.7.7/spec.html#json_encoding

Re: Using Protobuf instead of JSON to communicate with a front end

#57

Post by haberman » Fri, Jun 05, 2015, 5:20 PM UTC

oppositelock wrote: post #54

I worked on a product inside Google which used protos (v1) as the data format to a web front end, and in practice, that system was a failure, in part to the decision to use protos. The deserialization cost of protocol buffers is too high if you're doing complex data throughput, and even though the data size is smaller, it's better to send larger gzipped JSON (which will be decompressed in native code) and deserialize…

Things have changed a lot since your experience, I think. For one, a different encoding called "JSPB" has become the de facto standard for doing Protocol Buffers in JavaScript, at least inside Google. JSPB is parseable with JSON.parse(), so it avoids the speed issues you experienced.

And looking forward, JavaScript parsing of protobuf binary format has gotten a lot faster, thanks in large part to newer JavaScript technologies like TypedArray. Ideally JSPB would be deprecated as a wire format in favor of fast JavaScript parsing of binary protobufs, but this would of course be contingent on the performance being acceptable.

Finally, JSON is becoming a first-class citizen in proto3, so protobuf vs. JSON will no longer be an either/or, it can be a both/and. https://developers.google.com/protocol-buffers/docs/proto3#j...

Re: Using Protobuf instead of JSON to communicate with a front end

#58

Post by haberman » Fri, Jun 05, 2015, 5:24 PM UTC

caust1c wrote: post #52

haberman wrote: post #43

Making JSON first-class is an explicit design goal of proto3, the next version of Protocol Buffers currently in alpha: https://developers.google.com/protocol-buffers/docs/proto3#j... This will allow you to switch between JSON and protobuf binary on the wire easily, while using official protobuf client libraries. So you can choose easily whether you care more about size/speed efficiency or wire readability. Best of bo…

> Message field names are mapped to lowerCamelCase Why is a mapping to camel case necessary? I imagine it creates the potential for collisions, no?

I think this may be to maintain style conventions with JavaScript and previous versions of Google Cloud APIs which were all based in lowerCamelCase (not 100% sure though, don't quote me on this).

A benefit of this decision is that if you create JSON manually in JavaScript (ie. without a protobuf library) your JSON objects will match JavaScript conventions.

Re: Using Protobuf instead of JSON to communicate with a front end

#59

Post by shanemhansen » Fri, Jun 05, 2015, 5:41 PM UTC

rikrassen wrote: post #38

One of the comments on that article was "YAY! JSON is wastefully large. I'd love to replace it." Is this true? I'm confused why JSON would be seen as a wasteful as a format. It seems to be that with any decent compression I would think it's hard to get much smaller. In this case I'm not talking about the other advantages Protobuf offers, I just want to know about size.

There are basically 2 areas where JSON is really wasteful. Compression can help with both of those.

  1. Dictionary keys are repeated when you have an array of similar objects.
  2. Non-text data. JSON can't natively represent binary data, forcing people to use things like base64 for binary and base10 for numbers.

Re: Using Protobuf instead of JSON to communicate with a front end

#60

Post by boomzilla » Fri, Jun 05, 2015, 5:55 PM UTC

haberman wrote: post #57

oppositelock wrote: post #54

I worked on a product inside Google which used protos (v1) as the data format to a web front end, and in practice, that system was a failure, in part to the decision to use protos. The deserialization cost of protocol buffers is too high if you're doing complex data throughput, and even though the data size is smaller, it's better to send larger gzipped JSON (which will be decompressed in native code) and deserialize…

Things have changed a lot since your experience, I think. For one, a different encoding called "JSPB" has become the de facto standard for doing Protocol Buffers in JavaScript, at least inside Google. JSPB is parseable with JSON.parse(), so it avoids the speed issues you experienced. And looking forward, JavaScript parsing of protobuf binary format has gotten a lot faster, thanks in large part to newer JavaScript tec…

What benefits do I get from ProtoBuf, apart from the standard binary wire format?

JSON is just more popular as a serialization format. It doesn't matter what what programming language or OS I am on, there is almost always a built-in library that de/serialize JSONs at reasonable speed. To send the JSON objects around from one service to another, I can just gzip the string if it's big, or just plain UTF-8 string if it's not.

ProtoBuf has to provide more values for people like me to switch. I would rather try out Apache Avro first as a replacement for what I am doing right now.