Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

301–310 of 321 posts

Re: Protobuffers Are Wrong (2018)

#301

Post by cryptonector »

StopDisinfo910 wrote: post #289
cryptonector wrote: post #263

Earlier quoted context omitted.

This. Plus ASN.1 is pluggable as to encoding rules and has a large family of them: - BER/DER/CER (TLV) - OER and PER ("packed" -- no tags and no lengths wherever possible) - XER (XML!) - JER (JSON!) - GSER (textual representation) - you can add your own! (One could add one based on XDR, which would look a lot like OER/PER in a way.) ASN.1 also gives you a way to do things like formalize typed holes. Not looking at AS…

The people who wrote PB clearly knew ASN.1. It was the most famous IDL at the time. Do you assume they just came one morning and decided to write PB without taking a look at what existed? Anyway, as stated PB does more than ASN.1. It specifies both the description format and the encoding. PB is ready to be used out of the box. You have a compact IDL and a performant encoding format without having to think about anyth…

> Do you assume they just came one morning and decided to write PB without taking a look at what existed?

Considering how bad an imitation of 1984 ASN.1 PB's IDL is, and how bad an imitation of 1984 DER PB is, yes I assume that PB's creators did not in fact know ASN.1 well. They almost certainly knew of ASN.1, and they almost certainly did not know enough about it because all the worst mistakes in ASN.1 PB re-created while adding zero new ideas or functionality. It's a terrible shame.

Re: Protobuffers Are Wrong (2018)

#302

Post by cryptonector »

tliltocatl wrote: post #286
grogers wrote: post #269

Earlier quoted context omitted.

Other than ASN.1 PER, is there any other widely used encoding format that isn't self-describing? Using TLV certainly adds flexibility around schema evolution, but I feel like collectively we are wasting a fair amount of bytes because of it...

Cap'n'proto doesn't have tags, but it wastes even more bytes in favor of speed. Than again, omitting tags only saves space if you are sending all the fields every time. PER uses a bitmap, which is still a bit wasteful on large sparse structs.

PER sends a bitmap only of OPTIONAL members' (fields') presence/absence. Required members are just where you expect them: right after their preceding members.

Re: Protobuffers Are Wrong (2018)

#303

Post by kannanvijayan »

haberman wrote: post #99
zigzag312 wrote: post #79

Earlier quoted context omitted.

It looks similar to how vint64 lib encodes varints. Total length of varint can be determined via the first byte alone.

I advocated for PrefixVarint (which seems equivalent to vint64 ) for WebAssembly, but it was decided against, in favor of LEB128: https://github.com/WebAssembly/design/issues/601 The recent CREL format for ELF also uses the more established LEB128: https://news.ycombinator.com/item?id=41222021 At this point I don't feel like I have a clear opinion about whether PrefixVarint is worth it, compared with LEB128.

Varint encoding is something I've peeked at in various contexts. My personal bias is towards the prefix-style, as it feels faster to decode and the segregation of the meta-data from the payload data is nice.

But, the thing that tends to tip the scales is the fact that in almost all real world cases, small numbers dominate - as the github thread you linked relates in a comment.

The LEB128 fast-path is a single conditional with no data-dependencies:

  if ! (x & 0x80) { x }
Modern CPUs will characterize that branch really well and you'll pay almost zero cost for the fastpath which also happens to be the dominant path.

It's hard to beat.

Re: Protobuffers Are Wrong (2018)

#304

Post by instig007 »

kentonv wrote: post #300
instig007 wrote: post #295

Earlier quoted context omitted.

> Throwing a theorem-prover at the problem, unaided by developer hints, is not realistic in a large codebase. Please, Kenton, don't move your goalpost. Who said about "unaided"? Annotations, whether they come directly from a developer, or from IR meta, don't make a provided SAT-constraint suddenly a "dependent type" component of your type system, it needs a bit more than that. Let's not miss the "types" in "dependent…

Ah you are just trying to gaslight me. pytype doesn't do static bounds checking. What compels you to do this? Posting just to make people angry? Do you not have anything better to do with all that PL theory expertise?

> Ah you are just trying to gaslight me. pytype doesn't do static bounds checking.

It does static type checking from _annotations_ that live _outside_ the type system of the language. Have you forgotten that you began to argue that SMT solvers need constraint annotations to be realistic for static bounds checking in large codebases, and that the constraint annotations somehow become dependent types from that fact alone?

> What compels you to do this? Posting just to make people angry? Do you not have anything better to do with all that PL theory expertise?

You're all over the place, it's frustrating that instead of fairly addressing the points about inferior aspects of the protobuf protocol design that are unnecessary for the purpose of backward-compatible distributed systems, you keep saying (or at least assuming) that it's the only realistic solution, because "I worked at google" and "reports at google prove me right".

Re: Protobuffers Are Wrong (2018)

#305

Post by yencabulator »

bilkow wrote: post #232
sdenton4 wrote: post #224

Earlier quoted context omitted.

That's a nice idea... But I believe the design direction of proto buffers was to make everything `optional`, because `required` tends to bite you later when you realize it should actually be optional.

My understanding is that asymmetric fields provide a migration path in case that happens, as stated in the docs: > Unlike optional fields, an asymmetric field can safely be promoted to required and vice versa. > [...] > Suppose we now want to remove a required field. It may be unsafe to delete the field directly, since then clients might stop setting it before servers can handle its absence. But we can demote it to a…

> My understanding is that asymmetric fields provide a migration path in case that happens, as stated in the docs:

If you can assume you can churn a generation of fresh data soonish, and never again read the old data. For RPC sure, but someone like Google has petabytes of stored protobufs, so they don't pretend they can upgrade all the writers.

Re: Protobuffers Are Wrong (2018)

#306

Post by yencabulator »

kannanvijayan wrote: post #303
haberman wrote: post #99

Earlier quoted context omitted.

I advocated for PrefixVarint (which seems equivalent to vint64 ) for WebAssembly, but it was decided against, in favor of LEB128: https://github.com/WebAssembly/design/issues/601 The recent CREL format for ELF also uses the more established LEB128: https://news.ycombinator.com/item?id=41222021 At this point I don't feel like I have a clear opinion about whether PrefixVarint is worth it, compared with LEB128.

Varint encoding is something I've peeked at in various contexts. My personal bias is towards the prefix-style, as it feels faster to decode and the segregation of the meta-data from the payload data is nice. But, the thing that tends to tip the scales is the fact that in almost all real world cases, small numbers dominate - as the github thread you linked relates in a comment. The LEB128 fast-path is a single conditi…

SQLite format equivalent:

  if x
while strictly improving all other aspects (at least IMHO)

https://sqlite.org/src4/doc/trunk/www/varint.wiki

Re: Protobuffers Are Wrong (2018)

#307

Post by yencabulator »

usrnm wrote: post #168
leoc wrote: post #124

Earlier quoted context omitted.

What about Cap’n Proto https://capnproto.org/ ? (Don't know much about these things myself, but it's a name that usually comes up in these discussions.)

Cap'n'proto is not very nice to work with in C++, and I'd discourage anyone from using it from other programming languages, the implementations are just not there yet. We use both cnp and protobufs at work, and I vastly prefer protobufs, even for C++. I only wish they stayed the hell away from abseil, though.

The developer experience of capnproto is pretty darn miserable. I replaced my Rust use of it with https://rkyv.org/ -- probably the biggest ergonomic improvement was a single validation after which the message is safe to look at, instead of errors on every code path. The biggest downside was loss of built-in per-message schema evolution; in my use case I can have one version number up front.

Re: Protobuffers Are Wrong (2018)

#308

Post by [deleted] »

thethimble wrote: post #111
summerlight wrote: post #94

https://news.ycombinator.com/item?id=18190005 Just FYI: an obligatory comment from the protobuf v2 designer. Yeah, protobuf has lots of design mistakes but this article is written by someone who does not understand the problem space. Most of the complexity of serialization comes from implementation compatibility between different timepoints. This significantly limits design space.

Relatedly, most of the author's concerns are solved by wrapping things in a message. > oneof fields can’t be repeated. Wrap oneof field in message which can be repeated > map fields cannot be repeated. Wrap in message which can contain repeated fields > map values cannot be other maps. Wrap map in message which can be a value Perhaps this is slightly inconvenient/un-ergonomic, but the author is positioning these thin…

[deleted]

Re: Protobuffers Are Wrong (2018)

#309

Post by cryptonector »

grogers wrote: post #269
tliltocatl wrote: post #181

Earlier quoted context omitted.

Embedded/constrained UDP is where protobuf wire format (but not google's libraries) rocks: IoT over cellular and such, where you need to fit everything into a single datagram (number of roundtrips is what determines power consumption). As to those who say "UDP is unreliable" - what you do is you implement ARQ on the application level. Just like TCP does it, except you don't have to waste roundtrips on SYN-SYN-ACK han…

Other than ASN.1 PER, is there any other widely used encoding format that isn't self-describing? Using TLV certainly adds flexibility around schema evolution, but I feel like collectively we are wasting a fair amount of bytes because of it...

Also JSOON and XML are not TLV, though of course they're not really good examples of non-TLV encodings -- certainly they can't be what you had in mind.

Re: Protobuffers Are Wrong (2018)

#310

Post by StopDisinfo910 »

cryptonector wrote: post #301
StopDisinfo910 wrote: post #289

Earlier quoted context omitted.

The people who wrote PB clearly knew ASN.1. It was the most famous IDL at the time. Do you assume they just came one morning and decided to write PB without taking a look at what existed? Anyway, as stated PB does more than ASN.1. It specifies both the description format and the encoding. PB is ready to be used out of the box. You have a compact IDL and a performant encoding format without having to think about anyth…

> Do you assume they just came one morning and decided to write PB without taking a look at what existed? Considering how bad an imitation of 1984 ASN.1 PB's IDL is, and how bad an imitation of 1984 DER PB is, yes I assume that PB's creators did not in fact know ASN.1 well . They almost certainly knew of ASN.1, and they almost certainly did not know enough about it because all the worst mistakes in ASN.1 PB re-create…

PB is not a bad imitation of 1984 ASN.1. ASN.1 is choke full of useless representations clearly there to serve what a committee thought the need of the telco industry should be.

I find it funny you are making it looks like a good and pleasant to use IDL. It’s a perfect example of design by committee at its worst.

PB is significantly more space efficient than DER by the way.

Post reply on HN