Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protocol Buffers v3.0.0 released

github.com

Post reply on HN
Print view View original

31–40 of 131 posts

Re: Protocol Buffers v3.0.0 released

#31

Post by wwalser »

manish_gill wrote: post #12

If someone better informed than me can please explain - where and why would something like Protocol Buffers be useful?

Imagine working on a team that wants to move quickly but whose output is both a product and an API that's consumed by multiple other teams. The product you are building uses said API, but so do other teams. Your code needs to be stable enough to support these other teams needs (an API which doesn't change under them) but you also want to be able to make changes to your own application quickly, thus needing to change the API regularly.

A reasonable move is to version said API and have an ops team that ensures that all in-use versions of the API stay running. Some consumers will be on the bleeding edge, your team's application for example while others will lag behind.

Using proto* in this case is a reasonable move because you gain multiple benefits, performance being perhaps the least important in this case. Having a defined schema for your API provides some level of natural documentation for the API. Code generation allows your team to publish trusted client libraries for multiple languages.

I'll specifically call out client libraries since I've seen it make a dramatic difference in organizational efficiency, mostly to do with team to team trust levels. Without a client library the testing situation becomes a significant burden, read up on contract testing. When the team that's publishing an API also creates the client that most directly calls that API, the client library is the testing surface instead of every consumer of the API needing to test the API itself for regressions.

Re: Protocol Buffers v3.0.0 released

#32

Post by tantalor »

zellyn wrote: post #16

- removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. - the "well-known types" boxed primitive types essentially add optional values back in. And depending on your language bindings, may look the same. - extensions are still allowed in proto3 syntax files, but only for options - since the descriptor is still proto2. It seems odd to build a proto3 that…

> removing optional values

Slight correction: optional values are not removed. Quite the opposite; the "optional" keyword is removed because now all fields are optional. It is actually required values which were removed.

Re: Protocol Buffers v3.0.0 released

#33

Post by winstonewert »

merb wrote: post #29
prattmic wrote: post #10

Earlier quoted context omitted.

Within the API, proto3 does not have the concept of field presence. All fields are "present" and default to their type's zero value. Since the client can handle this, there is no need to explicitly serialize default values.

and how do you send a explicit zero so that the client knows that the field is really set by the server and not the default? or a explicit empty string?

If the client really needs that information, the server must explicitly include it in a seperate field.

Re: Protocol Buffers v3.0.0 released

#34

Post by NikhilVerma »

manish_gill wrote: post #12

If someone better informed than me can please explain - where and why would something like Protocol Buffers be useful?

At Badoo we use them to have a unified API for all of our platforms (Web, Mobile Web, Android, iOS, Windows Phone etc). This would not have been possible without something like ProtoBuf.

Re: Protocol Buffers v3.0.0 released

#35

Post by sanderjd »

manish_gill wrote: post #18
dkopi wrote: post #15

Earlier quoted context omitted.

> Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. From https://developers.google…

Yes, I read this. It tells me what Protocol Buffers are. Faster, Smaller XML like data structures for serialisation. What are the most common use cases though? And do people only use them for performance reasons?

The most common use cases line up with those of JSON: communication between programs that don't share an address space. The main advantage over JSON (in my opinion) is the definition of an explicit schema. The second (and also important) advantage is in the efficient size of the serialized data, which limits memory, disk, and bandwidth usage. Another (less important to me) advantage is in serialization and deserialization efficiency. A disadvantage is that it requires deserialization for human inspection - that is, it isn't plain text like JSON or XML.

It is similar to Apache Thrift, if you're looking for a non-Google project with similar ideas.

Re: Protocol Buffers v3.0.0 released

#36

Post by zellyn »

tantalor wrote: post #32
zellyn wrote: post #16

- removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. - the "well-known types" boxed primitive types essentially add optional values back in. And depending on your language bindings, may look the same. - extensions are still allowed in proto3 syntax files, but only for options - since the descriptor is still proto2. It seems odd to build a proto3 that…

> removing optional values Slight correction: optional values are not removed. Quite the opposite; the "optional" keyword is removed because now all fields are optional. It is actually required values which were removed.

True. But when using them, it feels like every field is "present" and you don't have to worry about the "optional, missing" case.

Re: Protocol Buffers v3.0.0 released

#37

Post by bmm6o »

wehadfun wrote: post #30

In C# why use Protocol Buffer over the XML or binary serializes?

Performance and data size are much better with protobufs: http://stackoverflow.com/questions/549128/fast-and-compact-o.... Built-in serializers are only workable when both ends are on the same platform (i.e. .Net), and even then class versioning can be a problem.

Re: Protocol Buffers v3.0.0 released

#38

Post by zellyn »

zellyn wrote: post #21
manish_gill wrote: post #12

If someone better informed than me can please explain - where and why would something like Protocol Buffers be useful?

We use them internally at Square for our RPC mechanism ("Sake", similar to "Stubby", Google's internal RPC mechanism), for our Kafka-based logging/metrics/queue infrastructure, and for defining external JSON APIs. We're in the process of switching from Sake to GRPC, which also use Protobufs as their payload format (although you can sub in different transports).

I should mention that we use Ruby, Java, and Go. So protobufs are also the "lingua franca" for cross-language communication.

Re: Protocol Buffers v3.0.0 released

#39

Post by tantalor »

merb wrote: post #29
prattmic wrote: post #10

Earlier quoted context omitted.

Within the API, proto3 does not have the concept of field presence. All fields are "present" and default to their type's zero value. Since the client can handle this, there is no need to explicitly serialize default values.

and how do you send a explicit zero so that the client knows that the field is really set by the server and not the default? or a explicit empty string?

One case where this question is important is when you are updating a record stored by the server. You only want to send fields you are changing because the record might be huge. But then how does the server distinguish between fields you didn't set and fields you want to set back to the default? The solution is to also tell the server which fields you are changing in a separate message.

Example:

    {
      'update_record': {
        # Set foo=bar
        'foo': 'bar'
      },
      'fields_to_update': {
        'foo': true,
        # Set some_int_flag=0 (default)
        'some_int_flag': true
      }
    }
See also "Field Masks in Update Operations" https://developers.google.com/protocol-buffers/docs/referenc...

Re: Protocol Buffers v3.0.0 released

#40

Post by blt »

I was hoping for packed serialization of non-primitive types. I once used Protobuf to serialize small point clouds, and ended up needing to serialize them as a packed double array and reconstruct the (x, y, z) structure at read time to avoid Protobuf malloc'ing each point individually. Not a huge deal, but it would be a real pain for more complex types.
Post reply on HN