Live data from Hacker News

Protocol Buffers v3.0.0 released

github.com

111–120 of 131 posts

Re: Protocol Buffers v3.0.0 released

#111
post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

This is only an issue if you use the Timestamp to represent a human-readable time. There are more uses for timestamping than for display to a human operator. For example, one might use a timestamp in a software system to detect the passage of time, as in the use of a monotonic clock. In a real-time system you would ignore the presence of leap seconds because you will never examine the timing of your system relative t…

I've always used uint64's for that. Why would you need a distinct type.

Re: Protocol Buffers v3.0.0 released

#112
post #64

How does this compare or in general why would you pick this vs newer formats like Cap'n'proto or FlatBuffers? From FlatBuffers overview I see this comparison: --- Protocol Buffers is indeed relatively similar to FlatBuffers, with the primary difference being that FlatBuffers does not need a parsing/ unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocatio…

Cap'n'proto is more or less abandoned I believe. But it and the flatbuffer approach gives very fast serialization and deserialization speed (essentially takes 0 times) but you pay a cost when you later access data, because it extracts the values you need on demand from the raw bytes. I'm not sure it would often make much sense overall.

What makes you think it is more or less abandoned?

Re: Protocol Buffers v3.0.0 released

#113
post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

This is only an issue if you use the Timestamp to represent a human-readable time. There are more uses for timestamping than for display to a human operator. For example, one might use a timestamp in a software system to detect the passage of time, as in the use of a monotonic clock. In a real-time system you would ignore the presence of leap seconds because you will never examine the timing of your system relative t…

If you use Google's timestamp type to burn for 250ms, you might end up with 250*86401/86400 ms. That's not a fantastic outcome.

Re: Protocol Buffers v3.0.0 released

#114
post #73

Earlier quoted context omitted.

In a trusted system, if you don't trust the structure you are working with, why would you trust the signature? I'd want to always work from the signed blob. That said, this is one reason to use flatbuffers/capt'n proto I guess: you don't have to worry about this since you never unpack the blob.

Think of a data flow A->B->C, with A e.g. handling incoming message server, B being a spam/virus filter, and C holding the user's mailbox. Spam/virus filters are useful, but are also rather vulnerable - so C is willing to trust B's spam/non-spam judgement, but wants to ensure that B can't alter or make up messages. If protobufs had one canonical encoding, B could unpack the message and re-pack it when done; with the…

So wouldn't you stick with the original message from A, and just have B sign that? You wouldn't want to have B repack it, because then B has the potential to muck with things.

Re: Protocol Buffers v3.0.0 released

#115
post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

This is only an issue if you use the Timestamp to represent a human-readable time. There are more uses for timestamping than for display to a human operator. For example, one might use a timestamp in a software system to detect the passage of time, as in the use of a monotonic clock. In a real-time system you would ignore the presence of leap seconds because you will never examine the timing of your system relative t…

I think you have it exactly backwards, if I understand things correctly.

It _seems_ like their "UTC Epoch time" is the same thing as POSIX time, but the Google engineer's terminology is all fubar. The reliance on the Proleptic Gregorian Calendar is further proof as that's a reference to a specific algorithm for calculating calendar dates.

POSIX time says that there are precisely 86400 "seconds" per day, which I think implies the same thing as saying there being precisely 60 "seconds" per minute. The logical consequence is, of course, that in neither case is "second" referring to the SI second.

Once you get over the fact that we're discussing different units of time, then you can see that POSIX time is _perfect_ for recording and manipulating civil calendar time. For the purposes of calendar manipulation, you rarely if ever need to know elapsed time in SI-unit seconds. All you care about is easily calculating past and _future_ calendar information. Your power company and credit card companies don't bill you by SI seconds, they bill you by the hour, day, week, or month.

Conversely, in those situations where you want accurate and precise SI-second measurements, you rarely if ever want to convert or display that data in terms of calendar time. When SpaceX sends a rocket into space, the view screen shows elapsed seconds since launch, not elapsed seconds since lunch. That's a big difference.

Interestingly, in neither case do leap seconds matter! They're irrelevant. Leaps second play no part in either TAI or POSIX time.

There are some cases where you want both pieces of information, but I think it's usually a mistake to conflate them and try to shoehorn them into the same units. That misguided practice is behind all the anxiety about leap seconds in UTC time.

It's also worth noting that as clocks become increasingly precise and accurate that the whole leap second thing will fade away. UTC time is based on the fiction that there's an abstract, universal clock in the world that is measurable in SI seconds. There isn't. At some point the needs of routine industrial measurements will enter the realm where relativity governs, at which point the fiction will be laid bare. Calendar time, of course, doesn't rely on that fiction.

The move to uncouple civil time from solar time is totally misguided, IMO, and only exacerbates the improper way that software engineers conflate the purpose and function of various time measurements.

Re: Protocol Buffers v3.0.0 released

#116

Earlier quoted context omitted.

> - removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. I feel the opposite; this greatly reduces the utility of protobuf. Previously, I could trust that if parsing succeeded, then I had a guarantee of a populated data structure. Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. That'…

> I could trust that if parsing succeeded, then I had a guarantee of a populated data structure Using required fields have actually bit Google more than once and were increasingly being considered harmful. A canonical example is that you add a required field, and then update binaryA in production (which receives messages from binaryB), which immediately crashes or errors out because the new field is missing. So pract…

> A canonical example is that you add a required field ...

Yeah. Don't do that without versioning your protocol. It's even less difficult to handle than maintaining API/ABI compatibility in a library.

> So practically speaking, you can never add required fields to any message where you can't guarantee binary version syncing amongst all instances of the message-dependent services.

Sure you can. If you version things at the protocol or per-request level, you can negotiate protocol conformance just fine.

Having a message type defined as "Message_V1" OR "Message_V2" is still simpler than having "any or none of the fields from any iteration of the message definition, where consistency is solely defined in terms of the field/message validation code you write in every protocol consumer".

> And if you're not running an RPC-based service architecture, then why are you using protos anyway?

It's a very serviceable compact serialization mechanism for at-rest data.

Re: Protocol Buffers v3.0.0 released

#117
post #70

Earlier quoted context omitted.

> - removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. I feel the opposite; this greatly reduces the utility of protobuf. Previously, I could trust that if parsing succeeded, then I had a guarantee of a populated data structure. Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. That'…

> Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. You always had to check the individual fields for the zero value. A required field in a proto2 message can be set but also have the default value and pass initialization.

> You always had to check the individual fields for the zero value.

No, you didn't. A required field has a value, period. If it defaults to a particular value, then that's the value it has.

If you had a non-required field, then you marked it 'optional', and checked for the field's existence (or mapped optional fields to a Maybe/Option monad representation, forcing the issue).

Re: Protocol Buffers v3.0.0 released

#118

Earlier quoted context omitted.

Could you expand on the problems you encountered?

I've never looked at proto3, but proto2 has at least the following issues: * No clue about namespacing. If you pick the wrong name for something, you can have name clashes within a protobuf, across uninterpreted option classes, with protobuf source code, with your own source code; and it's different if you're in Python or C. Nowhere are naming restrictions defined. * The API is maddening and inconsistent, especially…

The short answer is the Python implementation wasn't exactly great.

Re: Protocol Buffers v3.0.0 released

#119

Earlier quoted context omitted.

> I could trust that if parsing succeeded, then I had a guarantee of a populated data structure Using required fields have actually bit Google more than once and were increasingly being considered harmful. A canonical example is that you add a required field, and then update binaryA in production (which receives messages from binaryB), which immediately crashes or errors out because the new field is missing. So pract…

> A canonical example is that you add a required field ... Yeah. Don't do that without versioning your protocol. It's even less difficult to handle than maintaining API/ABI compatibility in a library. > So practically speaking, you can never add required fields to any message where you can't guarantee binary version syncing amongst all instances of the message-dependent services. Sure you can. If you version things a…

> Yeah. Don't do that without versioning your protocol. It's even less difficult to handle than maintaining API/ABI compatibility in a library.

Actually, the whole point of that was so you don't have to version your protocol. Protocol versioning actually tends to make code maintenance a pain in the posterior, and working through old data really annoying. Instead, you do optional fields.

If you don't want that, go ahead and just write raw bytes and don't bother with the serialization layer.

> Having a message type defined as "Message_V1" OR "Message_V2" is still simpler than having "any or none of the fields from any iteration of the message definition, where consistency is solely defined in terms of the field/message validation code you write in every protocol consumer".

But you don't have to do either. It seems like you aren't familiar with the use of protocol buffers. You just define optional fields with a reasonable default, and magically all the old protobufs get that default value.

Re: Protocol Buffers v3.0.0 released

#120

Earlier quoted context omitted.

> I could trust that if parsing succeeded, then I had a guarantee of a populated data structure Using required fields have actually bit Google more than once and were increasingly being considered harmful. A canonical example is that you add a required field, and then update binaryA in production (which receives messages from binaryB), which immediately crashes or errors out because the new field is missing. So pract…

> A canonical example is that you add a required field ... Yeah. Don't do that without versioning your protocol. It's even less difficult to handle than maintaining API/ABI compatibility in a library. > So practically speaking, you can never add required fields to any message where you can't guarantee binary version syncing amongst all instances of the message-dependent services. Sure you can. If you version things a…

> It's a very serviceable compact serialization mechanism for at-rest data.

That's fair, but then you run into the same issue -- adding required a field requires updating your entire store.

Depending on your store, that can range from onerous to outright impossible.

> Don't do that without versioning your protocol

I think it depends on your needs, but I think for most users, explicit versioning of messages is overkill and is just a more heavy way of encoding the same logic (e.g. I saw an older message, I will implicitly upgrade it by filling in these new fields, vs. just looking for the optional field that I just added)

Post reply on HN