Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Protobuf instead of JSON to communicate with a front end

blog.wearewizards.io

Post reply on HN
Print view View original

31–40 of 99 posts

Re: Using Protobuf instead of JSON to communicate with a front end

#31

Post by swalsh »

VikingCoder wrote: post #16
swalsh wrote: post #13

I always wondered why google decided to build Protocol Buffers. ASN.1 seemed like it worked well, and it covered all the corners.

Here was Kenton Varda's response: https://groups.google.com/forum/#!topic/protobuf/eNAZlnPKVW4 My understanding of ASN.1 is that it has no affordance for forwards- and backwards-compatibility, which is critical in distributed systems where the components are constantly changing. ... OK, I looked into this again (something I do once every few years when someone points it out). ASN.1 _by default_ has no extensibility,…

Fair enough.

Re: Using Protobuf instead of JSON to communicate with a front end

#32

Post by haberman »

cletus wrote: post #26
skybrian wrote: post #19

It's possible to encode a protobuf as JSON and we do it all the time at Google. In browsers, native JSON parsing is very fast and the data is compressed, so going to a binary format doesn't seem worthwhile. The .proto file is used basically as an IDL from which we generate code.

Personally I've found JSON encoded protobufs to be almost universally awful. The most common method is to use an array indexed by the field number. I've seen protobufs with hundreds of fields so that's hundreds of nulls as the string "null". The alternative is to have JSON objects with attributes named after the protobufs field name. This isn't without warts either and seems to be less prevelant in my experience. Ano…

> The most common method is to use an array indexed by the field number. I've seen protobufs with hundreds of fields so that's hundreds of nulls as the string "null".

What you are describing here is known as the "JSPB" wire format. This is a serialization that is only ever used for JavaScript, and only used there because, historically, parsing binary protobufs in JavaScript was too slow. With TypedArray and other JavaScript enhancements, this is changing. Ideally, JSPB wire format would be phased out completely.

> The alternative is to have JSON objects with attributes named after the protobufs field name. This isn't without warts either and seems to be less prevelant in my experience.

It's about to become a lot more prevalent with proto3, which features first-class JSON support. See: https://developers.google.com/protocol-buffers/docs/proto3#j...

Disclaimer: I work on the protobuf team.

Re: Using Protobuf instead of JSON to communicate with a front end

#33

Post by haberman »

hesdeadjim wrote: post #30
haberman wrote: post #28

Earlier quoted context omitted.

What you describe is exactly how proto3, the latest version of protobuf, will work! proto3 supports both binary protobuf encoding and JSON natively, so you can switch between them as desired. https://developers.google.com/protocol-buffers/docs/proto3 proto3 is currently in alpha, but we are working to bring it closer to release (I work on the protobuf team at Google).

Oh cool, didn't know there was a new version of protocol buffers. I ended up choosing Thrift for my current project due to wider language support, but I have been frustrated with some the limitations of the IDL (primarily no recursive data structures, so no generic storage of JSON-like objects).

Another one of the goals of proto3 is to increase language support a lot. Just this year we have added Ruby, Objective C, and C#, and keep your eyes peeled for more. proto3 is still alpha, but this is the direction things are going.

proto3 is especially designed to be paired with gRPC, which is also in alpha but also going for wide language support: http://www.grpc.io/

Re: Using Protobuf instead of JSON to communicate with a front end

#35

Post by jwr »

dustingetz wrote: post #2

Transit is similar but addresses the flaws described in this article http://blog.cognitect.com/blog/2014/7/22/transit

Transit is a really good solution.

As for Protobuf, I tried using it in a number of places, but found it to be very inflexible (schema!) and hard to debug in case of problems.

Re: Using Protobuf instead of JSON to communicate with a front end

#37

Post by benjaminjackman »

It would probably be better to try something like Cap'n Proto or SBE if worried about performance. Otherwise I think sticking to GZIP'd json isn't going to lag that far behind. Protocol buffers biggest benefit IMHO is just their .proto file for cross language code generation.

I have it on a todo list to port an SBE parser to ScalaJS. ScalaJS already backs java ByteBuffers with javascript TypedArrays. That should be really fast, the same stuff that is being worked on for making asm.js fast will also make the Cap'n Proto / SBE approach fast, so I think this has the most promise of bringing really high-performance data transfer capabilities to the browser.

Re: Using Protobuf instead of JSON to communicate with a front end

#38

Post by rikrassen »

One of the comments on that article was "YAY! JSON is wastefully large. I'd love to replace it." Is this true? I'm confused why JSON would be seen as a wasteful as a format. It seems to be that with any decent compression I would think it's hard to get much smaller. In this case I'm not talking about the other advantages Protobuf offers, I just want to know about size.

Re: Using Protobuf instead of JSON to communicate with a front end

#40

Post by maratd »

rikrassen wrote: post #38

One of the comments on that article was "YAY! JSON is wastefully large. I'd love to replace it." Is this true? I'm confused why JSON would be seen as a wasteful as a format. It seems to be that with any decent compression I would think it's hard to get much smaller. In this case I'm not talking about the other advantages Protobuf offers, I just want to know about size.

> Is this true?

No, it isn't true, but regardless of what format you use, there will always be someone who's not happy. Actually, I think that applies to everything in life.

Post reply on HN