Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protocol Buffers v3.0.0 released

github.com

Post reply on HN
Print view View original

101–110 of 131 posts

Re: Protocol Buffers v3.0.0 released

#101

Post by RyanZAG »

amluto wrote: post #47
alfalfasprout wrote: post #23

Earlier quoted context omitted.

Been using flatbuffers in production for a high speed market feed for a month now. Love it. Decode/encode time is absurdly fast (~1-2 microseconds for a small to medium schema). If you're pushing 50k+ events/second it can be a great choice. Takes up almost no space on the wire too.

Try Cap'n Proto instead. Better designed and faster.

Faster? To quote the author of Cap'n Proto:

> https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-...

"With that said, my intuition is that SBE will probably edge Cap’n Proto and FlatBuffers on performance in the average case, due to its decision to forgo support for random access. Between Cap’n Proto and FlatBuffers, it’s harder to say. FlatBuffers’ vtable approach seems like it would make access more expensive, though its simpler pointer format may be cheaper to follow. FlatBuffers also appears to do a lot of bookkeeping at encoding time which could get costly (such as de-duping vtables), but I don’t know how costly.

For most people, the performance difference is probably small enough that qualitative (feature) differences in the libraries matter more."

Re: Protocol Buffers v3.0.0 released

#102

Post by colanderman »

grosbisou wrote: post #3
mattiemass wrote: post #2

Wow, this seems to address a bunch of problems I've experienced with protobuf in the past. Looks awesome!

Could you expand on the problems you encountered?

I've never looked at proto3, but proto2 has at least the following issues:

* No clue about namespacing. If you pick the wrong name for something, you can have name clashes within a protobuf, across uninterpreted option classes, with protobuf source code, with your own source code; and it's different if you're in Python or C. Nowhere are naming restrictions defined.

* The API is maddening and inconsistent, especially in Python. (It's totally different between Python and C.) Some things look like lists but really aren't (e.g. you can't assign a list to a repeated field in Python). Even basic reflection (e.g. to get at uninterpreted options) is a Lovecraftian nightmare, and the docs are wholly unhelpful.

* Good luck serializing a list. There's not really such a thing, despite that the API pretends like there is; there are only repeated fields. So you need a separate flag to distinguish "empty list" from "not present list".

* Abstruse implementation. There are so many layers of indirection in the generated source and the core library that I wouldn't know where to start debugging.

Not sure if they fixed any of these issues with proto3.

Re: Protocol Buffers v3.0.0 released

#103

Post by smallnamespace »

slavik81 wrote: post #87
smallnamespace wrote: post #84

Earlier quoted context omitted.

> I could trust that if parsing succeeded, then I had a guarantee of a populated data structure Using required fields have actually bit Google more than once and were increasingly being considered harmful. A canonical example is that you add a required field, and then update binaryA in production (which receives messages from binaryB), which immediately crashes or errors out because the new field is missing. So pract…

For many applications, binary version sync is easy. In those cases, it does add unnecessary complexity.

Sure, if you can guarantee that your app will always be in that environment.

Otherwise, you run the risk of having to redo all your protos (+ downtime) if/when your app needs to scale up. I'm not sure whether that's worth avoiding some proto validation logic in the client code.

From https://developers.google.com/protocol-buffers/docs/proto#si...:

> Required Is Forever

> You should be very careful about marking fields as `required`. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field – old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some engineers at Google have come to the conclusion that using `required` does more harm than good; they prefer to use only `optional` and `repeated`. However, this view is not universal.

Re: Protocol Buffers v3.0.0 released

#104

Post by [deleted] »

merb wrote: post #29
prattmic wrote: post #10

Earlier quoted context omitted.

Within the API, proto3 does not have the concept of field presence. All fields are "present" and default to their type's zero value. Since the client can handle this, there is no need to explicitly serialize default values.

and how do you send a explicit zero so that the client knows that the field is really set by the server and not the default? or a explicit empty string?

[deleted]

Re: Protocol Buffers v3.0.0 released

#105

Post by mattiemass »

grosbisou wrote: post #3
mattiemass wrote: post #2

Wow, this seems to address a bunch of problems I've experienced with protobuf in the past. Looks awesome!

Could you expand on the problems you encountered?

Dealing with forward- and backwards-compatibility with enum changes has bit me many times in the past. So has required fields.

Re: Protocol Buffers v3.0.0 released

#106

Post by klodolph »

ithkuil wrote: post #53
klodolph wrote: post #46

Earlier quoted context omitted.

gRPC is based on Stubby: http://www.grpc.io/posts/principles

Stubby is based on protobuf; where by "based" I mean layered on, i.e. protobuf is the encoding used by stubby to encode requests and responses. gRPC is a reimplementation of Stubby suitable to be used outside of Google.

Yes, I think you are just using a different sense of "based on" than I am. gRPC is based on Stubby in the sense that it is influenced by the design of Stubby and uses the knowledge learned from creating Stubby.

Re: Protocol Buffers v3.0.0 released

#107

Post by andrewmcwatters »

euyyn wrote: post #93
andrewmcwatters wrote: post #83

Could someone explain to me why you would use Protocol Buffers, Cap'n Proto, etc versus rolling your own type-length-value protocol besides API interop? What if your team could write a smaller TLV protocol, and it was necessary to keep your codebase small? Would this not be wise? Are Protobufs and party not comparable to TLV protocols?

In the vast majority of cases, you want your team to spend their time doing something other than reinventing protos, debugging the in-house implementation, maintaining the library, etc. It's not clear to me anyway how doing it yourself would help keeping your codebase small vs using protos. In terms of code to maintain, doing it yourself is a net loss. In terms of binary size and method count, the proto libraries for…

Those are all reasons why I wanted to use protobufs to begin with. It sounded like it solved many issues for us.

But I'm thinking about scripting environments, where the data types used in protobufs don't exist in the host language. Simple things like this. I think in the implementations I've seen, they're just coerced or ignored. That's fine, imo.

But in terms of small codebases: a simple TLV protocol, where only limited data types are implemented, can be 1/10th of the size of any protobufs implementation.

My team has built out a high performance type-length-value system that doesn't require compiled schemas for game development, and we have a very small serialization lib that's smaller than any protobufs implementation for our target language.

I'd like to use protobufs to decrease the amount of modules we have to personally maintain, but I don't see the value in doing so for our particular situation.

Re: Protocol Buffers v3.0.0 released

#108

Post by ambrice »

skybrian wrote: post #92
ambrice wrote: post #74

Earlier quoted context omitted.

> Added a new field option "json_name". By default proto field names are converted to "lowerCamelCase" in proto3 JSON format. This option can be used to override this behavior and specify a different JSON name for the field.

Right, but nobody's going to set that for every single protobuf field.

You're right. The only people that would use it are people that a) care enough about optimization to switch out shorter tag names and b) don't care enough about optimization to switch to binary format. Probably not many..

Re: Protocol Buffers v3.0.0 released

#109

Post by icedchai »

amluto wrote: post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

It's a serialization format containing seconds and microseconds. You can put whatever you want in there, including true (non-Google) UTC time, right? This seems more like a documentation problem than an actual problem with Protobuf.

Re: Protocol Buffers v3.0.0 released

#110

Post by jhspaybar »

amluto wrote: post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

It saddens me that this is the top comment. It's complete and total FUD unrelated in any way to what Proto is, and to boot, it's an optional type, provided if you want it, but otherwise not forced to be used in any way! Scroll down the page for much more worthwhile discussions of Proto.
Post reply on HN