Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Protobuf instead of JSON to communicate with a front end

blog.wearewizards.io

Post reply on HN
Print view View original

61–70 of 99 posts

Re: Using Protobuf instead of JSON to communicate with a front end

#61

Post by haberman »

boomzilla wrote: post #60
haberman wrote: post #57

Earlier quoted context omitted.

Things have changed a lot since your experience, I think. For one, a different encoding called "JSPB" has become the de facto standard for doing Protocol Buffers in JavaScript, at least inside Google. JSPB is parseable with JSON.parse(), so it avoids the speed issues you experienced. And looking forward, JavaScript parsing of protobuf binary format has gotten a lot faster, thanks in large part to newer JavaScript tec…

What benefits do I get from ProtoBuf, apart from the standard binary wire format? JSON is just more popular as a serialization format. It doesn't matter what what programming language or OS I am on, there is almost always a built-in library that de/serialize JSONs at reasonable speed. To send the JSON objects around from one service to another, I can just gzip the string if it's big, or just plain UTF-8 string if it'…

In my opinion, the biggest benefit from using protobuf is that the schema exists in a .proto file. This can be used to provide all sorts of conveniences.

With a plain JSON-based API, you copy and paste field names out of sample code or the documentation. If you spell a field name wrong, there will be no error on the client. If you're lucky, the server might error out because it didn't recognize the property name, but it also might not. If you send an integer when the server was expecting a string, the server might automatically convert or it might not.

With protobuf, the schema is explicit in a .proto file. That means that the client library can tell you, at the precise moment that you say msg.misspledFieldName, that the field name doesn't exist. Or if you try to put an integer in there instead of a string, it can tell you about that too. Basically it makes for a tighter feedback loop, which is almost always better.

In statically-typed languages like C++ or Java, the schema can be used to generate static types too, so it's actually a compile-time error when you misspell a field name.

> It doesn't matter what what programming language or OS I am on, there is almost always a built-in library that de/serialize JSONs at reasonable speed.

Yep, that's one reason that proto3 will support JSON as a first-class citizen: https://developers.google.com/protocol-buffers/docs/proto3#j...

Re: Using Protobuf instead of JSON to communicate with a front end

#62

Post by skybrian »

teh wrote: post #20
skybrian wrote: post #19

It's possible to encode a protobuf as JSON and we do it all the time at Google. In browsers, native JSON parsing is very fast and the data is compressed, so going to a binary format doesn't seem worthwhile. The .proto file is used basically as an IDL from which we generate code.

Can I ask which library you are using? I found a few [1] but none seem super robust. Also, how do you deal with the bytes type? [1] https://code.google.com/p/protobuf-json/ https://github.com/benhodgson/protobuf-to-dict

There's no standard for this and it's mostly not open source as far as I know. The overall approach is called "JSPB" but there are various flavors. A typical use case is for a web app that has its own private RPC to its own servers, so interop isn't an issue. Also, web apps generally don't need or want to work with binary data, so better not to send it.

I recently became the maintainer of the Dart protobuf library which supports both JSON and binary format [1], [2]. However, the JSON format isn't necessarily compatible with other protobuf libraries you've seen.

[1] https://github.com/dart-lang/dart-protobuf [2] https://github.com/dart-lang/dart-protoc-plugin

Re: Using Protobuf instead of JSON to communicate with a front end

#63

Post by gcb0 »

haberman wrote: post #61
boomzilla wrote: post #60

Earlier quoted context omitted.

What benefits do I get from ProtoBuf, apart from the standard binary wire format? JSON is just more popular as a serialization format. It doesn't matter what what programming language or OS I am on, there is almost always a built-in library that de/serialize JSONs at reasonable speed. To send the JSON objects around from one service to another, I can just gzip the string if it's big, or just plain UTF-8 string if it'…

In my opinion, the biggest benefit from using protobuf is that the schema exists in a .proto file. This can be used to provide all sorts of conveniences. With a plain JSON-based API, you copy and paste field names out of sample code or the documentation. If you spell a field name wrong, there will be no error on the client. If you're lucky, the server might error out because it didn't recognize the property name, but…

summary: xml annoyances for json

:-)

Re: Using Protobuf instead of JSON to communicate with a front end

#64

Post by abecedarius »

haberman wrote: post #47
zapov wrote: post #45

Earlier quoted context omitted.

Do you plan on improving Protobuf speed in Java? People don't expect it to be slower than JSON ;) http://hperadin.github.io/jvm-serializers-report/report.html

I'm not super familiar with the Java implementation, but I believe it's pretty optimized and appears to do very well generally on that benchmark. One unavoidable issue is that, unlike JSON, protobuf serializers have to do two passes over the message tree, because in protobuf binary format all submessages are prefixed by their length. The first pass just calculates lengths, while the second performs the actual seriali…

So encode from back to front? Then when you reach the front you know the length.

Re: Using Protobuf instead of JSON to communicate with a front end

#65

Post by hosay123 »

cletus wrote: post #26
skybrian wrote: post #19

It's possible to encode a protobuf as JSON and we do it all the time at Google. In browsers, native JSON parsing is very fast and the data is compressed, so going to a binary format doesn't seem worthwhile. The .proto file is used basically as an IDL from which we generate code.

Personally I've found JSON encoded protobufs to be almost universally awful. The most common method is to use an array indexed by the field number. I've seen protobufs with hundreds of fields so that's hundreds of nulls as the string "null". The alternative is to have JSON objects with attributes named after the protobufs field name. This isn't without warts either and seems to be less prevelant in my experience. Ano…

I suspect you don't know what disclaimer means

Re: Using Protobuf instead of JSON to communicate with a front end

#66

Post by nostrademons »

boomzilla wrote: post #60
haberman wrote: post #57

Earlier quoted context omitted.

Things have changed a lot since your experience, I think. For one, a different encoding called "JSPB" has become the de facto standard for doing Protocol Buffers in JavaScript, at least inside Google. JSPB is parseable with JSON.parse(), so it avoids the speed issues you experienced. And looking forward, JavaScript parsing of protobuf binary format has gotten a lot faster, thanks in large part to newer JavaScript tec…

What benefits do I get from ProtoBuf, apart from the standard binary wire format? JSON is just more popular as a serialization format. It doesn't matter what what programming language or OS I am on, there is almost always a built-in library that de/serialize JSONs at reasonable speed. To send the JSON objects around from one service to another, I can just gzip the string if it's big, or just plain UTF-8 string if it'…

The encoding for protobufs is significantly more compact than JSON. If you're logging or persistently storing your data on the server, this can cut down your storage and bandwidth costs significantly, particularly if you're operating at Google scale.

Haberman also mentioned the schema benefits.

All that said, I'm using JSON for my current startup. I view them as optimizing for different parts of the product's lifecycle: JSON lets you quickly adapt the protocol and switch out different languages for different services when you're figuring out what product to build, while Protobuf saves you money when you're trying to scale it. I'm also pretty intrigued by Cap'n Proto as a high-performance serialization format, since it fixes a lot of the problems we faced using protobufs at scale at Google, but its language support just isn't up to protobuf/JSON yet, and the protocol is quite complicated.

Re: Using Protobuf instead of JSON to communicate with a front end

#67

Post by skybrian »

haberman wrote: post #32
cletus wrote: post #26

Earlier quoted context omitted.

Personally I've found JSON encoded protobufs to be almost universally awful. The most common method is to use an array indexed by the field number. I've seen protobufs with hundreds of fields so that's hundreds of nulls as the string "null". The alternative is to have JSON objects with attributes named after the protobufs field name. This isn't without warts either and seems to be less prevelant in my experience. Ano…

> The most common method is to use an array indexed by the field number. I've seen protobufs with hundreds of fields so that's hundreds of nulls as the string "null". What you are describing here is known as the "JSPB" wire format. This is a serialization that is only ever used for JavaScript, and only used there because, historically, parsing binary protobufs in JavaScript was too slow. With TypedArray and other Jav…

proto3's JSON is an improvement on ascii protobufs, but since it uses field names, it doesn't have the same backward compatibility guarantees as a format that uses tag numbers.

It would be nice if we had a standardized JSPB wire format that used tag numbers, rather than the various unofficial implementations we have now.

Re: Using Protobuf instead of JSON to communicate with a front end

#68

Post by haberman »

gcb0 wrote: post #63
haberman wrote: post #61

Earlier quoted context omitted.

In my opinion, the biggest benefit from using protobuf is that the schema exists in a .proto file. This can be used to provide all sorts of conveniences. With a plain JSON-based API, you copy and paste field names out of sample code or the documentation. If you spell a field name wrong, there will be no error on the client. If you're lucky, the server might error out because it didn't recognize the property name, but…

summary: xml annoyances for json :-)

XML isn't painful because it has a schema, XML is painful because it wasn't really designed for RPC, so getting to feature parity with something like Protocol Buffers takes a whole stack of XML technologies and a huge mess of complexity.

Protocol Buffers were designed from the ground up for RPC, and as a result are far simpler and more convenient to use than XML. Seriously, nobody who uses Protocol Buffers compares them to XML, because it's not even a comparison.

https://developers.google.com/protocol-buffers/docs/overview...

Re: Using Protobuf instead of JSON to communicate with a front end

#69

Post by iagooar »

caust1c wrote: post #52
haberman wrote: post #43

Making JSON first-class is an explicit design goal of proto3, the next version of Protocol Buffers currently in alpha: https://developers.google.com/protocol-buffers/docs/proto3#j... This will allow you to switch between JSON and protobuf binary on the wire easily, while using official protobuf client libraries. So you can choose easily whether you care more about size/speed efficiency or wire readability. Best of bo…

> Message field names are mapped to lowerCamelCase Why is a mapping to camel case necessary? I imagine it creates the potential for collisions, no?

It's called "sadCamelCase", as opposed to "HappyCamelCase" ;)

Re: Using Protobuf instead of JSON to communicate with a front end

#70

Post by sapek »

haberman wrote: post #43

Making JSON first-class is an explicit design goal of proto3, the next version of Protocol Buffers currently in alpha: https://developers.google.com/protocol-buffers/docs/proto3#j... This will allow you to switch between JSON and protobuf binary on the wire easily, while using official protobuf client libraries. So you can choose easily whether you care more about size/speed efficiency or wire readability. Best of bo…

JSON [1][2] (in fact text-based protocols in general) is also first-class in Bond (Microsoft's framework similar to ProtoBuf).

[1] https://microsoft.github.io/bond/manual/bond_cpp.html#simple...

[2] https://microsoft.github.io/bond/manual/bond_cs.html#json

Post reply on HN