Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Protobuf instead of JSON to communicate with a front end

blog.wearewizards.io

Post reply on HN
Print view View original

71–80 of 99 posts

Re: Using Protobuf instead of JSON to communicate with a front end

#71

Post by skybrian »

haberman wrote: post #57
oppositelock wrote: post #54

I worked on a product inside Google which used protos (v1) as the data format to a web front end, and in practice, that system was a failure, in part to the decision to use protos. The deserialization cost of protocol buffers is too high if you're doing complex data throughput, and even though the data size is smaller, it's better to send larger gzipped JSON (which will be decompressed in native code) and deserialize…

Things have changed a lot since your experience, I think. For one, a different encoding called "JSPB" has become the de facto standard for doing Protocol Buffers in JavaScript, at least inside Google. JSPB is parseable with JSON.parse(), so it avoids the speed issues you experienced. And looking forward, JavaScript parsing of protobuf binary format has gotten a lot faster, thanks in large part to newer JavaScript tec…

I'm not sure that TypedArray will help that much. For web apps, most of the data is strings and at some point you have to deserialize the strings so that regular JavaScript code can work with them (rather than asm.js code which would work with the bytes directly).

The proof of concept would be to send an array of strings as bytes in a TypedArray, deserialize it to an array of JavaScript strings using JavaScript (not native code), and show that this is about as fast as doing the same thing using JSON.parse(). It seems likely that JSON.parse() will have an easier time creating all those JavaScript strings and other objects at once from native code.

Re: Using Protobuf instead of JSON to communicate with a front end

#73

Post by [deleted] »

abecedarius wrote: post #64
haberman wrote: post #47

Earlier quoted context omitted.

I'm not super familiar with the Java implementation, but I believe it's pretty optimized and appears to do very well generally on that benchmark. One unavoidable issue is that, unlike JSON, protobuf serializers have to do two passes over the message tree, because in protobuf binary format all submessages are prefixed by their length. The first pass just calculates lengths, while the second performs the actual seriali…

So encode from back to front? Then when you reach the front you know the length.

[deleted]

Re: Using Protobuf instead of JSON to communicate with a front end

#74

Post by mhahn »

nostrademons wrote: post #66
boomzilla wrote: post #60

Earlier quoted context omitted.

What benefits do I get from ProtoBuf, apart from the standard binary wire format? JSON is just more popular as a serialization format. It doesn't matter what what programming language or OS I am on, there is almost always a built-in library that de/serialize JSONs at reasonable speed. To send the JSON objects around from one service to another, I can just gzip the string if it's big, or just plain UTF-8 string if it'…

The encoding for protobufs is significantly more compact than JSON. If you're logging or persistently storing your data on the server, this can cut down your storage and bandwidth costs significantly, particularly if you're operating at Google scale. Haberman also mentioned the schema benefits. All that said, I'm using JSON for my current startup. I view them as optimizing for different parts of the product's lifecyc…

I use Protobufs for my startup and it has saved us an incredible amount of time building out iOS, Android, and Web clients. With a small team, any time we can shave by not having to re-write the modeling layer in all of these languages is a big win. As the writer of the APIs, I publish the new Protobuf models/services and then can switch over and instantly start working with real objects in Swift or Java.

Coming from a larger startup, I've also experienced the pains of trying to maintain JSON objects between different services. Protobufs have some quirks, but I think its a great solution to get behind at any stage.

Re: Using Protobuf instead of JSON to communicate with a front end

#75

Post by nostrademons »

krapht wrote: post #72

How does Protobuf compare with Corba? I'd be interested in anybody's experience if they have used both.

CORBA was ridiculously complex, because they tried to make remote objects look like local ones, with messages, reference counting, naming, discovery, etc. Protobuf is just a serialization mechanism. You're thinking at a lower level of abstraction - it's all just PODs that go over the wire, you build your own RPC framework on top of that (or use gRPC, which is Google's protobuf-over-HTTP2 RPC library) and think in terms of requests & responses.

IMHO trying to make everything look like an object was a mistake, and newer RPC frameworks like gRPC, Thrift, and JSON-over-HTTP are much easier to use than the late-90s frameworks like RMI, CORBA, and DCOM. Sometimes you don't want abstraction, because it abstracts away details you absolutely need to think about.

Re: Using Protobuf instead of JSON to communicate with a front end

#76

Post by mcav »

skybrian wrote: post #71
haberman wrote: post #57

Earlier quoted context omitted.

Things have changed a lot since your experience, I think. For one, a different encoding called "JSPB" has become the de facto standard for doing Protocol Buffers in JavaScript, at least inside Google. JSPB is parseable with JSON.parse(), so it avoids the speed issues you experienced. And looking forward, JavaScript parsing of protobuf binary format has gotten a lot faster, thanks in large part to newer JavaScript tec…

I'm not sure that TypedArray will help that much. For web apps, most of the data is strings and at some point you have to deserialize the strings so that regular JavaScript code can work with them (rather than asm.js code which would work with the bytes directly). The proof of concept would be to send an array of strings as bytes in a TypedArray, deserialize it to an array of JavaScript strings using JavaScript (not…

TextDecoder/TextEncoder is the emerging standard way to do this.

https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder

Re: Using Protobuf instead of JSON to communicate with a front end

#78

Post by alkonaut »

rikrassen wrote: post #38

One of the comments on that article was "YAY! JSON is wastefully large. I'd love to replace it." Is this true? I'm confused why JSON would be seen as a wasteful as a format. It seems to be that with any decent compression I would think it's hard to get much smaller. In this case I'm not talking about the other advantages Protobuf offers, I just want to know about size.

> "YAY! JSON is wastefully large. I'd love to replace it." Is this true? I'm confused why JSON would be seen as a wasteful as a format.

It transmits type and field names. Depending on how complex your data is those strings could be a large part of the data.

{ "person": { "age": 30, "shoesize": 10 } }

The above is what, 4-5 bytes of protobuf? I'm not sure what the gzipped-json data is but likely a lot more. If you were to send a list of 100 such person objects, the difference would be smaller.

Re: Using Protobuf instead of JSON to communicate with a front end

#79

Post by haberman »

mcav wrote: post #76
skybrian wrote: post #71

Earlier quoted context omitted.

I'm not sure that TypedArray will help that much. For web apps, most of the data is strings and at some point you have to deserialize the strings so that regular JavaScript code can work with them (rather than asm.js code which would work with the bytes directly). The proof of concept would be to send an array of strings as bytes in a TypedArray, deserialize it to an array of JavaScript strings using JavaScript (not…

TextDecoder/TextEncoder is the emerging standard way to do this. https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder

Interesting, how does that relate to this? https://encoding.spec.whatwg.org/

Re: Using Protobuf instead of JSON to communicate with a front end

#80

Post by haberman »

abecedarius wrote: post #64
haberman wrote: post #47

Earlier quoted context omitted.

I'm not super familiar with the Java implementation, but I believe it's pretty optimized and appears to do very well generally on that benchmark. One unavoidable issue is that, unlike JSON, protobuf serializers have to do two passes over the message tree, because in protobuf binary format all submessages are prefixed by their length. The first pass just calculates lengths, while the second performs the actual seriali…

So encode from back to front? Then when you reach the front you know the length.

Interesting idea, and could be interesting to experiment with.

There are a lot of practical challenges though -- to provide a useful API you'd have to reverse the string at the end, since socket APIs don't generally provide streaming "WriteReverse" functions, and for good reason, because it would force them to buffer arbitrary amounts of data.

So we'd have to reverse the entire string at the end. The question is whether this would be cheaper than doing a second pass over the message tree. And also keep in mind that you would need the first pass to decode everything -- including UTF-8 data -- in reverse. But since the UTF-8 APIs for Java strings probably don't support this, you'd probably have to encode it, then reverse it to put it in the encoding buffer. That way when it gets reversed again at the end, it would be proper UTF-8.

At the end of the day, this probably wouldn't end up faster than what we do now. But can't say for sure without trying!

Post reply on HN