Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protocol Buffers v3.0.0 released

github.com

Post reply on HN
Print view View original

71–80 of 131 posts

Re: Protocol Buffers v3.0.0 released

#71

Post by cbsmith »

teacup50 wrote: post #58
zellyn wrote: post #16

- removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. - the "well-known types" boxed primitive types essentially add optional values back in. And depending on your language bindings, may look the same. - extensions are still allowed in proto3 syntax files, but only for options - since the descriptor is still proto2. It seems odd to build a proto3 that…

> - removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. I feel the opposite; this greatly reduces the utility of protobuf. Previously, I could trust that if parsing succeeded, then I had a guarantee of a populated data structure. Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. That'…

In practice, to have decent compatibility as revisions changed, you really had to minimize the use of "required" fields anyway. While I agree it was sometimes nice to be able to avoid having to worry about it, in practice protobuf parsing imposes a very minimal set of constraints on data types. A successful protobuf parse was not nearly enough to ensure you had data integrity. I've run in to more than a few cases of developers using the wrong protobuf (v2) definition and not realizing their successful parse was still wrong.

Re: Protocol Buffers v3.0.0 released

#72

Post by cbsmith »

madgar wrote: post #70
teacup50 wrote: post #58

Earlier quoted context omitted.

> - removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. I feel the opposite; this greatly reduces the utility of protobuf. Previously, I could trust that if parsing succeeded, then I had a guarantee of a populated data structure. Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. That'…

> Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. You always had to check the individual fields for the zero value. A required field in a proto2 message can be set but also have the default value and pass initialization.

I think you wording is a bit difficult to parse. Does this convert what you are trying to say:

In proto2, default values are for optional fields. The value of an optional field would be that value, but there's a separate concern as to whether that field had been set.

Re: Protocol Buffers v3.0.0 released

#73

Post by cbsmith »

JoachimSchipper wrote: post #4

This looks like a nice evolution. It's a pity that the "deterministic serialization" gives so few guarantees; I have worked on at least one project that really needed this. (Basically, we wanted to parse a signed blob, do some work, and pass the original data on without breaking the signature; unfortunately, this requires keeping the serialized form around, since the serialized form cannot be re-generated from its pa…

In a trusted system, if you don't trust the structure you are working with, why would you trust the signature?

I'd want to always work from the signed blob.

That said, this is one reason to use flatbuffers/capt'n proto I guess: you don't have to worry about this since you never unpack the blob.

Re: Protocol Buffers v3.0.0 released

#74

Post by ambrice »

skybrian wrote: post #48
positr0n wrote: post #17

Earlier quoted context omitted.

Why isn't is suitable? (I've never used protobufs)

The fields are indexed by field names (converted to lower camel case) instead of tag numbers. It's great for readability, but it's a lot more verbose, particularly for repeated fields.

> Added a new field option "json_name". By default proto field names are converted to "lowerCamelCase" in proto3 JSON format. This option can be used to override this behavior and specify a different JSON name for the field.

Re: Protocol Buffers v3.0.0 released

#75

Post by jschwartzi »

amluto wrote: post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

This is only an issue if you use the Timestamp to represent a human-readable time. There are more uses for timestamping than for display to a human operator. For example, one might use a timestamp in a software system to detect the passage of time, as in the use of a monotonic clock. In a real-time system you would ignore the presence of leap seconds because you will never examine the timing of your system relative to a Gregorian calendar. Rather, you just want to make sure that the station-keeping engine on your satellite burns for exactly 250 milliseconds, and leap seconds are of no use in that application.

Re: Protocol Buffers v3.0.0 released

#76

Post by amluto »

justinsaccount wrote: post #56
amluto wrote: post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

It's interesting that you refer to a huge amount of planning and engineering as "sticking your head in the sand". https://googleblog.blogspot.com/2011/09/time-technology-and-... I think that the approach everything else uses is the "sticking your head in the sand approach". You basically pretend that there is no problem and that time is perfectly accurate, up until you have a minute with 59 or 61 seconds. Just becaus…

No, I agree they did a bunch of good engineering for internal use.

But they didn't keep it internal properly -- the real world has leap seconds for better or for worse, and this library really does stick its head in the sand and pretend they don't exist. Google specifically says that this library is designed to be "the foundation of Google's new API platform". Yet they give a data type (as a headline feature) and a sample usage that is simply incorrect if you don't set your system to work using Google's "leap smear". It also seems quite likely that it'll result in blatantly wrong human-readable strings. I'll even quote a string from timestamp.proto [1]:

9999-12-31T23:59:59Z

That looks like an RFC 3339 string, and it even has the 'Z' suffix, which means it's UTC, which has an agreed-upon international definition. But this is not a valid UTC time. It's a time in a different time zone that Google made up.

Google easily could have done better: publish a spec for a different kind of time like:

9999-12-31T23:59:59s

where the little 's' means 'smeared'. Supply a serializer and deserializer for that. Now there's no ambiguity.

[1] https://github.com/google/protobuf/blob/master/src/google/pr...

Re: Protocol Buffers v3.0.0 released

#77

Post by artursapek »

skybrian wrote: post #49
the_duke wrote: post #42

Earlier quoted context omitted.

Why would you use JSON in a high performance context anyway?

Because the code is running in a browser. Browsers do support binary data using typed arrays but for some reason this isn't commonly used yet. Compatibility, maybe?

This library is very good at serializing and deserializing binary proto3 messages in the browser.

https://github.com/dcodeIO/ProtoBuf.js/

Re: Protocol Buffers v3.0.0 released

#78

Post by wongarsu »

amluto wrote: post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

It's fine as a timestamp implementation, and great for many uses. But I think a big problem with the documentation. They start off by saying it's "at nanosecond resolution in UTC Epoch time", and then they go on to explain how it uses a completely different encoding that is neither compatible with UTC nor with TAI (atomic time which ignores leap seconds). And then they jump ahead to sample code which again pretends that the timestamp is UTC.

No matter whether you like "google time" or not, this is horrible documentation. They are glossing over an issue which should be marked with big red letters.

Re: Protocol Buffers v3.0.0 released

#79

Post by pherl »

JoachimSchipper wrote: post #4

This looks like a nice evolution. It's a pity that the "deterministic serialization" gives so few guarantees; I have worked on at least one project that really needed this. (Basically, we wanted to parse a signed blob, do some work, and pass the original data on without breaking the signature; unfortunately, this requires keeping the serialized form around, since the serialized form cannot be re-generated from its pa…

The main concern that the deterministic serialization isn't canonical is due to the unknown fields. As string and message type share the same wire type, when parsing an unknown string/message type, the parser has no idea whether to recursively canonicalize the unknown field.

The cross-language inconsistency is mainly due to the string fields comparison performance, i.e. java/objc uses utf16 encodings which has different orderings than utf8 strings due to surrogate pairs.

Feel free to start an issue on the github site asking for canonical serialization with your use case. We may change the deterministic serialization with stronger guarantee (e.g. cross language consistency) or add another API for canonical serialization.

Re: Protocol Buffers v3.0.0 released

#80

Post by pherl »

zbjornson wrote: post #54
tantalor wrote: post #39

Earlier quoted context omitted.

One case where this question is important is when you are updating a record stored by the server. You only want to send fields you are changing because the record might be huge. But then how does the server distinguish between fields you didn't set and fields you want to set back to the default? The solution is to also tell the server which fields you are changing in a separate message. Example: { 'update_record': {…

Thanks for explaining that and for the reference. It seems like a lot of overhead for a protocol that is designed to be cheap...

There are also wrapper well known types that you can fallback to (in wrappers.proto), when you need to distinguish between, say empty string and null.
Post reply on HN