Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protocol Buffers v3.0.0 released

github.com

Post reply on HN
Print view View original

61–70 of 131 posts

Re: Protocol Buffers v3.0.0 released

#61

Post by peq »

teacup50 wrote: post #58
zellyn wrote: post #16

- removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. - the "well-known types" boxed primitive types essentially add optional values back in. And depending on your language bindings, may look the same. - extensions are still allowed in proto3 syntax files, but only for options - since the descriptor is still proto2. It seems odd to build a proto3 that…

> - removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. I feel the opposite; this greatly reduces the utility of protobuf. Previously, I could trust that if parsing succeeded, then I had a guarantee of a populated data structure. Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. That'…

I agree. In particular in languages without null you will have a lot of Option types in the mapping. You no longer can generate useful type definitions from the proto spec.

Re: Protocol Buffers v3.0.0 released

#62

Post by vtalwar »

NikhilVerma wrote: post #34
manish_gill wrote: post #12

If someone better informed than me can please explain - where and why would something like Protocol Buffers be useful?

At Badoo we use them to have a unified API for all of our platforms (Web, Mobile Web, Android, iOS, Windows Phone etc). This would not have been possible without something like ProtoBuf.

Nikhil, would love to know more about your architecture.

Re: Protocol Buffers v3.0.0 released

#63

Post by sigil »

manish_gill wrote: post #12

If someone better informed than me can please explain - where and why would something like Protocol Buffers be useful?

I'll give you an example.

I used protobuf as the output format for a web crawler. Workers read urls and sequentially write entire HTTP responses to disk. [0] Sure, you could serialize the responses to JSON, but the overhead of representing things like binary image data as escaped unicode strings was prohibitive in my case.

"Why not BSON?" Well, schemas can be nice when performance matters. Instead of solving a parsing problem at runtime, a C/C++ reader can contain a compiler-optimized deserializer for a given protobuf schema. It's almost like directly reading and writing an array of C structs, except protobuf is architecture-independent, and you can add new fields without breaking old readers.

There are plenty of reasons to not use protobuf. I particularly disliked the code generation step for C/C++. That makes even less sense in a language like Python, and yet that's exactly what the official python protobuf implementation from Google does (did?). I wrote a python protobuf library on top of a C protobuf library that avoids codegen: https://github.com/acg/lwpb

[0] See the ARC format used by the Internet Archive for a similar (and imo clunkier) solution. http://crawler.archive.org/articles/developer_manual/arcs.ht...

Re: Protocol Buffers v3.0.0 released

#64

Post by rdtsc »

How does this compare or in general why would you pick this vs newer formats like Cap'n'proto or FlatBuffers?

From FlatBuffers overview I see this comparison:

---

Protocol Buffers is indeed relatively similar to FlatBuffers, with the primary difference being that FlatBuffers does not need a parsing/ unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocation. The code is an order of magnitude bigger, too. Protocol Buffers has neither optional text import/export nor schema language features like unions.

---

So are the newer ones useful mostly when serialization vs deserialization speed matters (https://google.github.io/flatbuffers/) ?

Re: Protocol Buffers v3.0.0 released

#65

Post by peq »

skybrian wrote: post #48
positr0n wrote: post #17

Earlier quoted context omitted.

Why isn't is suitable? (I've never used protobufs)

The fields are indexed by field names (converted to lower camel case) instead of tag numbers. It's great for readability, but it's a lot more verbose, particularly for repeated fields.

I think the only reason to use json over the binary format is readability, so why care about this?

Re: Protocol Buffers v3.0.0 released

#66

Post by cbsmith »

rdtsc wrote: post #64

How does this compare or in general why would you pick this vs newer formats like Cap'n'proto or FlatBuffers? From FlatBuffers overview I see this comparison: --- Protocol Buffers is indeed relatively similar to FlatBuffers, with the primary difference being that FlatBuffers does not need a parsing/ unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocatio…

Also when you want to memory map a file/have live objects in shared memory, or in general have your in-memory & serialized structures be the same.

Re: Protocol Buffers v3.0.0 released

#67

Post by [deleted] »

justinsaccount wrote: post #56
amluto wrote: post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

It's interesting that you refer to a huge amount of planning and engineering as "sticking your head in the sand". https://googleblog.blogspot.com/2011/09/time-technology-and-... I think that the approach everything else uses is the "sticking your head in the sand approach". You basically pretend that there is no problem and that time is perfectly accurate, up until you have a minute with 59 or 61 seconds. Just becaus…

[deleted]

Re: Protocol Buffers v3.0.0 released

#68

Post by rybosome »

manish_gill wrote: post #18
dkopi wrote: post #15

Earlier quoted context omitted.

> Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. From https://developers.google…

Yes, I read this. It tells me what Protocol Buffers are. Faster, Smaller XML like data structures for serialisation. What are the most common use cases though? And do people only use them for performance reasons?

Performance is a nice benefit, but the standardization of message passing is by far the biggest benefit in my opinion. Within a given language, I know that any API I call will have certain unvarying semantics, I can see a highly readable yet formal spec of the data being exchanged, and the code to manipulate these messages will always be familiar and idiomatic.

Duplicating these benefits with XML or JSON would require defining your own grammar and parser, but wouldn't have the performance benefits. Recreating the performance gains would require a new serialization scheme, at which point you'd have broken from JSON and XML standard tools and recreated protobufs in everything but the proto definition language; at that point, why not create a DSL rather than bolting this functionality into an existing one?

Re: Protocol Buffers v3.0.0 released

#69

Post by cbsmith »

amluto wrote: post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

There's really no reason you can't provide your own timestamp structure, or your own timestamp transformation logic...

Re: Protocol Buffers v3.0.0 released

#70

Post by madgar »

teacup50 wrote: post #58
zellyn wrote: post #16

- removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. - the "well-known types" boxed primitive types essentially add optional values back in. And depending on your language bindings, may look the same. - extensions are still allowed in proto3 syntax files, but only for options - since the descriptor is still proto2. It seems odd to build a proto3 that…

> - removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. I feel the opposite; this greatly reduces the utility of protobuf. Previously, I could trust that if parsing succeeded, then I had a guarantee of a populated data structure. Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. That'…

> Now, I have to check each field individually, in manually written code, to verify that no required fields are missing.

You always had to check the individual fields for the zero value. A required field in a proto2 message can be set but also have the default value and pass initialization.

Post reply on HN