Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Protobuf instead of JSON to communicate with a front end

blog.wearewizards.io

Post reply on HN
Print view View original

81–90 of 99 posts

Re: Using Protobuf instead of JSON to communicate with a front end

#81

Post by haberman »

alkonaut wrote: post #78
rikrassen wrote: post #38

One of the comments on that article was "YAY! JSON is wastefully large. I'd love to replace it." Is this true? I'm confused why JSON would be seen as a wasteful as a format. It seems to be that with any decent compression I would think it's hard to get much smaller. In this case I'm not talking about the other advantages Protobuf offers, I just want to know about size.

> "YAY! JSON is wastefully large. I'd love to replace it." Is this true? I'm confused why JSON would be seen as a wasteful as a format. It transmits type and field names. Depending on how complex your data is those strings could be a large part of the data. { "person": { "age": 30, "shoesize": 10 } } The above is what, 4-5 bytes of protobuf? I'm not sure what the gzipped-json data is but likely a lot more. If you wer…

> The above is what, 4-5 bytes of protobuf?

Assuming the integer fields are regular varint types (and not the "fixed" integer encoding), and assuming the tag numbers were all under 16, then this would be a six-byte protobuf.

Re: Using Protobuf instead of JSON to communicate with a front end

#82

Post by teacup50 »

haberman wrote: post #43

Making JSON first-class is an explicit design goal of proto3, the next version of Protocol Buffers currently in alpha: https://developers.google.com/protocol-buffers/docs/proto3#j... This will allow you to switch between JSON and protobuf binary on the wire easily, while using official protobuf client libraries. So you can choose easily whether you care more about size/speed efficiency or wire readability. Best of bo…

I'm really bummed that you got rid of required fields in pb3. Now every consumer has to write additional code to verify that their required fields are actually available, and the proto spec is barely useful as an actual interpretable spec -- you have to specify requirements purely in comments.

On top of which, you've defined built-in default values for empty fields; this means that, without warning, an accidentally missing field will inject bad data into any consumer that doesn't carefully check for the existence of all required fields.

These are basically killer issues for us; we're not going to adopt an "update" that requires us to write JSON-style "hey, does this field exist?" code everywhere.

Re: Using Protobuf instead of JSON to communicate with a front end

#83

Post by alkonaut »

haberman wrote: post #81
alkonaut wrote: post #78

Earlier quoted context omitted.

> "YAY! JSON is wastefully large. I'd love to replace it." Is this true? I'm confused why JSON would be seen as a wasteful as a format. It transmits type and field names. Depending on how complex your data is those strings could be a large part of the data. { "person": { "age": 30, "shoesize": 10 } } The above is what, 4-5 bytes of protobuf? I'm not sure what the gzipped-json data is but likely a lot more. If you wer…

> The above is what, 4-5 bytes of protobuf? Assuming the integer fields are regular varint types (and not the "fixed" integer encoding), and assuming the tag numbers were all under 16, then this would be a six-byte protobuf.

Thanks. Admittedly a favourable example for protobufs but it shows the point: the names aren't data.

Re: Using Protobuf instead of JSON to communicate with a front end

#84

Post by haberman »

alkonaut wrote: post #83
haberman wrote: post #81

Earlier quoted context omitted.

> The above is what, 4-5 bytes of protobuf? Assuming the integer fields are regular varint types (and not the "fixed" integer encoding), and assuming the tag numbers were all under 16, then this would be a six-byte protobuf.

Thanks. Admittedly a favourable example for protobufs but it shows the point: the names aren't data.

Yep! This improves decoding speed too.

Re: Using Protobuf instead of JSON to communicate with a front end

#85

Post by Keats »

flavor8 wrote: post #77

> Reading time: ~15 minutes. 842 words including code. Average adult reading speed: 300 words/minute. Does not compute.

I know, I included some time for people wanting to to open some links, the github project etc.

Only reading the text itself takes indeed less than 5 minutes, not sure which approach people prefer.

Re: Using Protobuf instead of JSON to communicate with a front end

#86

Post by rqebmm »

Having used both on a few projects, including a JS frontend, my advice is:

"Don't use protobufs if you don't have to".

Protobufs can be much faster, and provide a strict schema, but it comes at the price of higher maintenance costs. JSON is much simpler, easier to implement, and MUCH easier to debug. If your GPB looks like it's building properly, but fails to parse, it's a huge pain to try and decode/debug the binary. You'll wish you could just print the JSON string.

If you need the speed and schema, then GPBs are great. In our case, we got a huge speed boost just by avoiding string building/parsing inherent in JSON.

Re: Using Protobuf instead of JSON to communicate with a front end

#88

Post by kentonv »

VikingCoder wrote: post #16
swalsh wrote: post #13

I always wondered why google decided to build Protocol Buffers. ASN.1 seemed like it worked well, and it covered all the corners.

Here was Kenton Varda's response: https://groups.google.com/forum/#!topic/protobuf/eNAZlnPKVW4 My understanding of ASN.1 is that it has no affordance for forwards- and backwards-compatibility, which is critical in distributed systems where the components are constantly changing. ... OK, I looked into this again (something I do once every few years when someone points it out). ASN.1 _by default_ has no extensibility,…

Man that guy sounds full of himself.

Ugh was that only 5 years ago?

Re: Using Protobuf instead of JSON to communicate with a front end

#89

Post by abecedarius »

haberman wrote: post #80
abecedarius wrote: post #64

Earlier quoted context omitted.

So encode from back to front? Then when you reach the front you know the length.

Interesting idea, and could be interesting to experiment with. There are a lot of practical challenges though -- to provide a useful API you'd have to reverse the string at the end, since socket APIs don't generally provide streaming "WriteReverse" functions, and for good reason, because it would force them to buffer arbitrary amounts of data. So we'd have to reverse the entire string at the end. The question is whet…

Reversing the output didn't come up when I did this sort of thing in C (for a different format): you can fill a buffer from the end, then just write the tail. But I guess Java does stick you with that? And I didn't have to deal with encoding to UTF-8 (ouch), or whatever other complications you might face, like streaming. Oh well! Hope the suggestion was stimulating anyway.

Re: Using Protobuf instead of JSON to communicate with a front end

#90

Post by nfmangano »

rqebmm wrote: post #86

Having used both on a few projects, including a JS frontend, my advice is: "Don't use protobufs if you don't have to". Protobufs can be much faster, and provide a strict schema, but it comes at the price of higher maintenance costs. JSON is much simpler, easier to implement, and MUCH easier to debug. If your GPB looks like it's building properly, but fails to parse, it's a huge pain to try and decode/debug the binary…

Could you elaborate on the maintenance costs? We use ProtoBufjs for our own real-time whiteboarding webapp over web sockets, and in the long run having strict schemas has saved us a lot of time. We're a distributed team with different members working on the front and backends, and we frequently refer to our proto files to remember how data is transferred and how it should be interpreted (explained in our proto commented code).

Are the maintenance costs related to debugging unparsable messages? We've almost never had an issue there, so maybe we've just been lucky?

Post reply on HN