Live data from Hacker News

Protocol Buffers v3.0.0 released

github.com

81–90 of 131 posts

Re: Protocol Buffers v3.0.0 released

#81
post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

It's interesting that you refer to a huge amount of planning and engineering as "sticking your head in the sand". https://googleblog.blogspot.com/2011/09/time-technology-and-... I think that the approach everything else uses is the "sticking your head in the sand approach". You basically pretend that there is no problem and that time is perfectly accurate, up until you have a minute with 59 or 61 seconds. Just becaus…

>You basically pretend that there is no problem and that time is perfectly accurate, up until you have a minute with 59 or 61 seconds.

Time is perfectly accurate, including all the minutes with 59 or 61 seconds. UTC is perfectly defined as atomic time (TAI) with an offset to keep it within 0.9 seconds of UT1 (time as measured by the rotation of the earth). Every time we increment or decrement this offset, this leads to leap seconds. But since 23:59:60 is a valid time (and distinct from 00:00:00 on days with leap seconds), there is no ambiguity here.

The problem here is how most computers handle this: introducing ambiguity by setting the clock backwards or forwards one second, instead of accounting for the fact that not all minutes have 60 seconds. Google did a pragmatic fix for their use case by squeezing leap seconds into the surrounding seconds, stretching them. It works for them, but now their "seconds" are not actual seconds anymore.

Re: Protocol Buffers v3.0.0 released

#82
post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

Regarding the leaking of NTP, are you talking about Systemd's default pointing at Google's NTP servers or some other event?

Re: Protocol Buffers v3.0.0 released

#83
Could someone explain to me why you would use Protocol Buffers, Cap'n Proto, etc versus rolling your own type-length-value protocol besides API interop?

What if your team could write a smaller TLV protocol, and it was necessary to keep your codebase small? Would this not be wise? Are Protobufs and party not comparable to TLV protocols?

Re: Protocol Buffers v3.0.0 released

#84
post #16

- removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. - the "well-known types" boxed primitive types essentially add optional values back in. And depending on your language bindings, may look the same. - extensions are still allowed in proto3 syntax files, but only for options - since the descriptor is still proto2. It seems odd to build a proto3 that…

> - removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. I feel the opposite; this greatly reduces the utility of protobuf. Previously, I could trust that if parsing succeeded, then I had a guarantee of a populated data structure. Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. That'…

> I could trust that if parsing succeeded, then I had a guarantee of a populated data structure

Using required fields have actually bit Google more than once and were increasingly being considered harmful.

A canonical example is that you add a required field, and then update binaryA in production (which receives messages from binaryB), which immediately crashes or errors out because the new field is missing.

So practically speaking, you can never add required fields to any message where you can't guarantee binary version syncing amongst all instances of the message-dependent services. At scale, this is essentially operationally impossible.

And if you're not running an RPC-based service architecture, then why are you using protos anyway?

Re: Protocol Buffers v3.0.0 released

#85
post #15

Earlier quoted context omitted.

> Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. From https://developers.google…

Yes, I read this. It tells me what Protocol Buffers are. Faster, Smaller XML like data structures for serialisation. What are the most common use cases though? And do people only use them for performance reasons?

RPC. Streaming.

Re: Protocol Buffers v3.0.0 released

#86
post #80

Earlier quoted context omitted.

Thanks for explaining that and for the reference. It seems like a lot of overhead for a protocol that is designed to be cheap...

There are also wrapper well known types that you can fallback to (in wrappers.proto), when you need to distinguish between, say empty string and null.

https://github.com/google/protobuf/blob/master/src/google/pr...

Re: Protocol Buffers v3.0.0 released

#87

Earlier quoted context omitted.

> - removing optional values is actually quite nice. In practice, I end up checking for "missing or empty string" anyway. I feel the opposite; this greatly reduces the utility of protobuf. Previously, I could trust that if parsing succeeded, then I had a guarantee of a populated data structure. Now, I have to check each field individually, in manually written code, to verify that no required fields are missing. That'…

> I could trust that if parsing succeeded, then I had a guarantee of a populated data structure Using required fields have actually bit Google more than once and were increasingly being considered harmful. A canonical example is that you add a required field, and then update binaryA in production (which receives messages from binaryB), which immediately crashes or errors out because the new field is missing. So pract…

For many applications, binary version sync is easy. In those cases, it does add unnecessary complexity.

Re: Protocol Buffers v3.0.0 released

#88

Earlier quoted context omitted.

Yes, I read this. It tells me what Protocol Buffers are. Faster, Smaller XML like data structures for serialisation. What are the most common use cases though? And do people only use them for performance reasons?

The most common use cases line up with those of JSON: communication between programs that don't share an address space. The main advantage over JSON (in my opinion) is the definition of an explicit schema. The second (and also important) advantage is in the efficient size of the serialized data, which limits memory, disk, and bandwidth usage. Another (less important to me) advantage is in serialization and deserializ…

Serialization and deserialization efficiency is specially important for mobile apps, in which the CPU used to parse/serialize JSON (or gzipped JSON) can become very prominent.

Apache Thrift, IIRC, is actually a reimplementation of protos, in the same way that Facebook's Buck is of Google's Bazel.

I have some times looked at "raw" binary protos to inspect the string fields, that happen(ed?) to be byte-aligned and so readable in a text editor. Not sure off the top of my head if that's always the case.

Re: Protocol Buffers v3.0.0 released

#89
post #64

How does this compare or in general why would you pick this vs newer formats like Cap'n'proto or FlatBuffers? From FlatBuffers overview I see this comparison: --- Protocol Buffers is indeed relatively similar to FlatBuffers, with the primary difference being that FlatBuffers does not need a parsing/ unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocatio…

Cap'n'proto is more or less abandoned I believe. But it and the flatbuffer approach gives very fast serialization and deserialization speed (essentially takes 0 times) but you pay a cost when you later access data, because it extracts the values you need on demand from the raw bytes.

I'm not sure it would often make much sense overall.

Re: Protocol Buffers v3.0.0 released

#90
post #45

They added a feature that impressively fails to interoperate with the rest of the world. > Added well-known type protos (any.proto, empty.proto, timestamp.proto, duration.proto, etc.). Users can import and use these protos just like regular proto files. Additional runtime support are available for each language. From timestamp.proto: // A Timestamp represents a point in time independent of any time zone // or calenda…

I'm glad they're willing to break compatibility to push their approach, because I think it's a better one. UTC with leap seconds is the worst of all possible worlds - not suitable for human time, not suitable for system time either - as perennial leap second bugs in such high-profile projects as the linux kernel demonstrate. Everyone seems to have agreed for years that basing system time on something without leap seconds would be better - whether that be leap smears or TAI - but no-one bothers to take action.
Post reply on HN