Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

291–300 of 321 posts

Re: Protobuffers Are Wrong (2018)

#291

Post by instig007 »

kentonv wrote: post #248
instig007 wrote: post #207

Earlier quoted context omitted.

> Real-world practice has also shown that quite often, fields that originally seemed to be "required" turn out to be optional over time how often? as practiced by who, and where? > 2. You actually do not want a oneof field to be repeated! > How do you make this change without breaking compatibility? Now you wish that you had defined your array as an array of messages, each containing a oneof, so that you could add a…

> how often? as practiced by who, and where? This was my experience in Google Search infrastructure circa 2005-2010. This was a system with dozens of teams and hundreds of developers all pushing their data through a common message bus. It happened all the damned time and caused multiple real outages (from overzealous validation), along with a lot of tech debt involving having to initialize fields with dummy data beca…

> This was my experience in Google Search infrastructure circa 2005-2010 [...]

> Reports from other large teams at google

> teach everyone how that works, etc.

> Or we could just tell people to wrap the thing in a `message`

It really sounds like a self-inflicted internal google issue. Can you address the part where I mention isomorphism of (oneof token) and (oneof (token {})), and clarify what exactly do you think you'd have to teach other engineers to do, if your protocol's encoders and decoders took this property into account?

Re: Protobuffers Are Wrong (2018)

#292

Post by instig007 »

tptacek wrote: post #219
instig007 wrote: post #178

Earlier quoted context omitted.

Yeah, let's pretend that type algebra doesn't exist, and even if it does exist then it's not useful and definitely isn't practical in data protocols. Let's believe that the authors of protobuf considered everything, and since they aren't amateurs (by the virtue of having worked on protobuf at Google, presumably), every elaborated opinion that draws them as amateurs at applying type algebra in data protocol designs is…

They're not amateurs by virtue of being some of the most senior engineers ever to work at Google. You don't get to play the "ad hominem" card while calling them names. This whole thread is embarrassing.

Ok, "some of the most senior engineers ever to work at Google" don't seem to know that static bounds checking don't require dependent types: https://news.ycombinator.com/item?id=45150008

> You don't get to play the "ad hominem" card while calling them names

The entire article explains it at length why there's the impression, it's not ad-hominem.

Re: Protobuffers Are Wrong (2018)

#293

Post by kentonv »

instig007 wrote: post #290
kentonv wrote: post #215

Earlier quoted context omitted.

> type algebra either doesn't exist or impractical because only PL theorists know about it, not Kenton. Hi I'm Kenton. I, too, was enamored with advanced PL theory in college. Designed and implemented my own purely-functional programming language. Still wish someone would figure out a working version of dependent types for real-world use, mainly so we could prove array bounds-safety without runtime checks. In two dec…

> Still wish someone would figure out a working version of dependent types for real-world use, mainly so we could prove array bounds-safety without runtime checks. Hi Kenton, I'm not sure what kind of PL theory you studied in college, but "array bounds-safety without runtime checks" don't require dependent types. They are being proven with several available SMT solvers as of right now, just ask LLVM folks with their…

Throwing a theorem-prover at the problem, unaided by developer hints, is not realistic in a large codebase. You need annotations that let you say "this array's size is is the same as that array" or "this integer is within the bounds of that array" -- that's dependent types.

Re: Protobuffers Are Wrong (2018)

#294

Post by mkoubaa »

usrnm wrote: post #223
mkoubaa wrote: post #196

Earlier quoted context omitted.

I'd rather they just used the abseil headers they needed with the abseil license at the top than make it a build dependency. The concept of a package is antithetical to C++ and no amount of tooling can fix that.

abseil is not header-only, though

Skill issue

Re: Protobuffers Are Wrong (2018)

#295

Post by instig007 »

kentonv wrote: post #293
instig007 wrote: post #290

Earlier quoted context omitted.

> Still wish someone would figure out a working version of dependent types for real-world use, mainly so we could prove array bounds-safety without runtime checks. Hi Kenton, I'm not sure what kind of PL theory you studied in college, but "array bounds-safety without runtime checks" don't require dependent types. They are being proven with several available SMT solvers as of right now, just ask LLVM folks with their…

Throwing a theorem-prover at the problem, unaided by developer hints, is not realistic in a large codebase. You need annotations that let you say "this array's size is is the same as that array" or "this integer is within the bounds of that array" -- that's dependent types.

> Throwing a theorem-prover at the problem, unaided by developer hints, is not realistic in a large codebase.

Please, Kenton, don't move your goalpost. Who said about "unaided"? Annotations, whether they come directly from a developer, or from IR meta, don't make a provided SAT-constraint suddenly a "dependent type" component of your type system, it needs a bit more than that. Let's not miss the "types" in "dependent types". You don't modify type systems of your languages to run SAT solvers in large codebases.

Truly, if you believe that annotations for the purpose of static bounds checking "is not realistic in a large codebase" (or is it because you assume it's unaided?), I've got "google/pytype" and the entire Python community to justify before you.

Re: Protobuffers Are Wrong (2018)

#296

Post by troupo »

StopDisinfo910 wrote: post #289
cryptonector wrote: post #263

Earlier quoted context omitted.

This. Plus ASN.1 is pluggable as to encoding rules and has a large family of them: - BER/DER/CER (TLV) - OER and PER ("packed" -- no tags and no lengths wherever possible) - XER (XML!) - JER (JSON!) - GSER (textual representation) - you can add your own! (One could add one based on XDR, which would look a lot like OER/PER in a way.) ASN.1 also gives you a way to do things like formalize typed holes. Not looking at AS…

The people who wrote PB clearly knew ASN.1. It was the most famous IDL at the time. Do you assume they just came one morning and decided to write PB without taking a look at what existed? Anyway, as stated PB does more than ASN.1. It specifies both the description format and the encoding. PB is ready to be used out of the box. You have a compact IDL and a performant encoding format without having to think about anyth…

> The people who wrote PB clearly knew ASN.1.

And your assumption is based on what exactly?

> It was the most famous IDL at the time.

Strange that at the same time (2001) people were busy implementing everyting in Java and XML, not ASN.1

> Do you assume they just came one morning and decided to write PB without taking a look at what existed?

Yes, that is a great assumption. Looking at what most companies do, this is an assumption bordering on prescience.

Re: Protobuffers Are Wrong (2018)

#297

Post by kentonv »

instig007 wrote: post #291
kentonv wrote: post #248

Earlier quoted context omitted.

> how often? as practiced by who, and where? This was my experience in Google Search infrastructure circa 2005-2010. This was a system with dozens of teams and hundreds of developers all pushing their data through a common message bus. It happened all the damned time and caused multiple real outages (from overzealous validation), along with a lot of tech debt involving having to initialize fields with dummy data beca…

> This was my experience in Google Search infrastructure circa 2005-2010 [...] > Reports from other large teams at google > teach everyone how that works, etc. > Or we could just tell people to wrap the thing in a `message` It really sounds like a self-inflicted internal google issue. Can you address the part where I mention isomorphism of (oneof token) and (oneof (token {})), and clarify what exactly do you think yo…

You seem to have merged the required fields issue and the oneof issue, but these are unrelated threads.

> Can you address the part where I mention isomorphism of (oneof token) and (oneof (token {})), and clarify what exactly do you think you'd have to teach other engineers to do, if your protocol's encoders and decoders took this property into account?

What you have written is not a serious proposal in terms of a working way to extend Protocol Buffers to allow repeated oneofs.

What you have written is a very complicated way of saying: "You theoretically could support extensible repeated oneofs, with the right type system and protocol design."

Yes, I know that. With a clean slate, we can do anything. But in the real world (yes I'm going to keep saying that, since you don't seem very familiar with it), you don't get to start from a clean slate every time you don't like how things have turned out.

As it stands, the product type in protobufs is `message`, the sum type is `oneof`, and the vector type is `repeated`. The way `oneof` is encoded on the wire is exactly one of the tags appear. The way `repeated` is encoded on the wire is that the same tag appears many times. The way `message` is encoded is that it's a length-delimited byte blob that contains a series of tag-values inside. Unfortunately, this encoding means if we supported `repeated oneof`, it would not be extensible.

So we ban `repeated oneof`, and say "you need to write a repeated message, where the message contains a `oneof`". This isn't as pretty as people might like but it works just fine in practice and we move on to more important things.

Re: Protobuffers Are Wrong (2018)

#298

Post by tptacek »

instig007 wrote: post #292
tptacek wrote: post #219

Earlier quoted context omitted.

They're not amateurs by virtue of being some of the most senior engineers ever to work at Google. You don't get to play the "ad hominem" card while calling them names. This whole thread is embarrassing.

Ok, "some of the most senior engineers ever to work at Google" don't seem to know that static bounds checking don't require dependent types: https://news.ycombinator.com/item?id=45150008 > You don't get to play the "ad hominem" card while calling them names The entire article explains it at length why there's the impression, it's not ad-hominem.

Previous threads on this story have spelled out specifically which Googlers were behind this design, and, again, it's embarrassing that anybody is trying to defend the hill of "protobuf's designers were amateurs". You can keep digging in if you want.

Re: Protobuffers Are Wrong (2018)

#299

Post by instig007 »

kentonv wrote: post #297
instig007 wrote: post #291

Earlier quoted context omitted.

> This was my experience in Google Search infrastructure circa 2005-2010 [...] > Reports from other large teams at google > teach everyone how that works, etc. > Or we could just tell people to wrap the thing in a `message` It really sounds like a self-inflicted internal google issue. Can you address the part where I mention isomorphism of (oneof token) and (oneof (token {})), and clarify what exactly do you think yo…

You seem to have merged the required fields issue and the oneof issue, but these are unrelated threads. > Can you address the part where I mention isomorphism of (oneof token) and (oneof (token {})), and clarify what exactly do you think you'd have to teach other engineers to do, if your protocol's encoders and decoders took this property into account? What you have written is not a serious proposal in terms of a wor…

> What you have written is not a serious proposal in terms of a working way to extend Protocol Buffers to allow repeated oneofs.

I didn't intend to propose a solution for protobuf specifically, I explained why the author of the subject article had a point in calling the authors of protobuf amateurs, given the existing spec, that led to specific implementations of the parsers, and the respective downsides.

> But in the real world (yes I'm going to keep saying that, since you don't seem very familiar with it)

I'll have to repeat that "real-world vs the rest of you" talking point is the specific attitude of (ex-)google folks that make them look amateur or, at least, ignorant.

> This isn't as pretty as people might like but it works just fine in practice and we move on to more important things.

That doesn't explain why you didn't implement it differtently, you just stated that you did. So, why didn't you implement it differently, if you admit that a few lines above with: "Yes, I know that. With a clean slate, we can do anything."

Re: Protobuffers Are Wrong (2018)

#300

Post by kentonv »

instig007 wrote: post #295
kentonv wrote: post #293

Earlier quoted context omitted.

Throwing a theorem-prover at the problem, unaided by developer hints, is not realistic in a large codebase. You need annotations that let you say "this array's size is is the same as that array" or "this integer is within the bounds of that array" -- that's dependent types.

> Throwing a theorem-prover at the problem, unaided by developer hints, is not realistic in a large codebase. Please, Kenton, don't move your goalpost. Who said about "unaided"? Annotations, whether they come directly from a developer, or from IR meta, don't make a provided SAT-constraint suddenly a "dependent type" component of your type system, it needs a bit more than that. Let's not miss the "types" in "dependent…

Ah you are just trying to gaslight me. pytype doesn't do static bounds checking.

What compels you to do this? Posting just to make people angry? Do you not have anything better to do with all that PL theory expertise?

Post reply on HN