Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

281–290 of 321 posts

Re: Protobuffers Are Wrong (2018)

#281

Post by lelanthran »

xg15 wrote: post #190

I'm starting to wonder if some of those bad design decisions are symptoms of a larger "cultural bias" at Google. Specifically the "No Compositionality" point: It reminds me of similar bad designs in Go, CSS and the web platform at large. The pattern seems to be that generalized, user-composable solutions are discouraged in favor of a myriad of special constructs that satisfy whatever concrete use cases seem relevant…

> This works for a while and reduces the complexity of the language upfront, while delivering results - but over time, the designs devolve into a rats's nest of hyperspecific design features with awkward and unintuitive restrictions.

But that's true for almost anything, though.

Re: Protobuffers Are Wrong (2018)

#282

Post by motorest »

ardit33 wrote: post #279
motorest wrote: post #257

Earlier quoted context omitted.

> JSON exists (...) The blog post leads with the personal assertion that "ad-hoc and built by amateurs". Therefore I doubt that JSON, a data serialization language designed by trimming most of JavaScript out and to be parses with eval(), would meet the opinionated high bar. Also, JSON is a data interchange language, and has no support for types beyond the notoriously ill-defined primitives. In contrast, protobuf is a…

You must be young. XML and XML Schemas existed before JSON or Protobuf, and people ditched them for a good reason and JSON took over. Protobuf is just another version of the old RPC/Java Beans, etc... of a binary format. Yes, it is more efficient data wise than JSON, but it is a PITA to work on and debug with.

> You must be young. XML and XML Schemas existed before JSON or Protobuf, and people ditched them for a good reason and JSON took over.

I'm not sure you got the point. It's irrelevant how old JSON or XML (a non sequitur) are. The point is that one of the main features and selling points of protobuf is strong typing and model validation implemented at the parsing level. JSON does not support any of these, and you need to onboard more than one ad-hoc tool to have a shot at feature parity, which goes against the blogger's opinionated position on the topic.

Re: Protobuffers Are Wrong (2018)

#283

Post by sethammons »

OrangeDelonge wrote: post #274
sethammons wrote: post #167

Earlier quoted context omitted.

We got bit by a default value in a DMS task where the target column didn't exist so the data wasn't replicated and the default value was "this work needs to be done." This is not pb nor go. A sensible default of invalid state would have caught this. So would an error and crash. Either would have been better than corrupt data.

You mean aws dms insterted the string literal “this work needs to be done” into your db?

So, that target column was called the wrong name, meaning data intended for the column never arrived, causing the default value in the database to be used, which was an integer that mapped to "this work item needs to be processed still" which led to double processing the record post dms migration

Re: Protobuffers Are Wrong (2018)

#284

Post by oftenwrong »

naikrovek wrote: post #276
lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

TLV style binary formats are all you need. The “Type” in that acronym is a 32-bit number which you can use to version all of your stuff so that files are backwards compatible. Software that reads these should read all versions of a particular type and write only the latest version. Code for TLV is easy to write and to read, which makes viewing programs easy. TLV data is fast for computers to write and to read. Protob…

Protobuf is typically serialised using a TLV-style encoding.

https://protobuf.dev/programming-guides/encoding/

A major value of protobuf is in its ecosystem of tools (codegen, lint, etc); it's not only an encoding. And you don't generally have to build or maintain any of it yourself, since it already exists and has significant industry investment.

Re: Protobuffers Are Wrong (2018)

#285

Post by lukaslalinsky »

I recently made a realization, that I can use MessagePack with a static schema defined in the code, and even pre-defined numeric field IDs, essentially replacing Protobuf for my use cases. I saw MessagePack as an alternative for JSON, with loose message structure, but it's actually a nice binary format and can be used more effectively than that. So now I enjoy things like tagged unions (in Zig/Python), and other types that are awkward to express in Protobuf. I settled on single character field names, for compatibility with msgspec, and I'm pretty happy with it. Still super compact messages with predictable schema, that are fast to parse, because I know which fields to expect.

Re: Protobuffers Are Wrong (2018)

#286

Post by tliltocatl »

grogers wrote: post #269
tliltocatl wrote: post #181

Earlier quoted context omitted.

Embedded/constrained UDP is where protobuf wire format (but not google's libraries) rocks: IoT over cellular and such, where you need to fit everything into a single datagram (number of roundtrips is what determines power consumption). As to those who say "UDP is unreliable" - what you do is you implement ARQ on the application level. Just like TCP does it, except you don't have to waste roundtrips on SYN-SYN-ACK han…

Other than ASN.1 PER, is there any other widely used encoding format that isn't self-describing? Using TLV certainly adds flexibility around schema evolution, but I feel like collectively we are wasting a fair amount of bytes because of it...

Cap'n'proto doesn't have tags, but it wastes even more bytes in favor of speed. Than again, omitting tags only saves space if you are sending all the fields every time. PER uses a bitmap, which is still a bit wasteful on large sparse structs.

Re: Protobuffers Are Wrong (2018)

#287

Post by kentonv »

lostdog wrote: post #273
kentonv wrote: post #253

Earlier quoted context omitted.

> It gets super frustrating to have to empty/null check fields everywhere you use them, especially for fields that are effectively required for the message to make sense. That's why Protobuf and Cap'n Proto have default values. You should not bother checking for presence of fields that are always supposed to be there. If the sender forgot to set a field, then they get the default value. That's their problem. > just a…

Sadly, the default values are an even bigger source of bugs. We just caught another one at $work where a field was never being filled in, but the default values made it look fine. It caused hidden failures later on. It's an incredibly frustrating "feature" to deal with, and causes lots of problems in proto3.

You can still verify presence explicitly if you want, with the `has` methods.

But if you don't check, it should return a default value rather than null. You don't want your server to crash on bad input.

Re: Protobuffers Are Wrong (2018)

#288

Post by cryptonector »

grogers wrote: post #269
tliltocatl wrote: post #181

Earlier quoted context omitted.

Embedded/constrained UDP is where protobuf wire format (but not google's libraries) rocks: IoT over cellular and such, where you need to fit everything into a single datagram (number of roundtrips is what determines power consumption). As to those who say "UDP is unreliable" - what you do is you implement ARQ on the application level. Just like TCP does it, except you don't have to waste roundtrips on SYN-SYN-ACK han…

Other than ASN.1 PER, is there any other widely used encoding format that isn't self-describing? Using TLV certainly adds flexibility around schema evolution, but I feel like collectively we are wasting a fair amount of bytes because of it...

OER (related to PER)

XDR (ONC RPC, NFS)

MS RPC (DCE RPC w/ tweaks)

Flat Buffers

Re: Protobuffers Are Wrong (2018)

#289

Post by StopDisinfo910 »

cryptonector wrote: post #263
jnwatson wrote: post #32

Earlier quoted context omitted.

ASN.1 implements message versioning in an extremely precise way. Implementing a linter would be trivial.

This. Plus ASN.1 is pluggable as to encoding rules and has a large family of them: - BER/DER/CER (TLV) - OER and PER ("packed" -- no tags and no lengths wherever possible) - XER (XML!) - JER (JSON!) - GSER (textual representation) - you can add your own! (One could add one based on XDR, which would look a lot like OER/PER in a way.) ASN.1 also gives you a way to do things like formalize typed holes. Not looking at AS…

The people who wrote PB clearly knew ASN.1. It was the most famous IDL at the time. Do you assume they just came one morning and decided to write PB without taking a look at what existed?

Anyway, as stated PB does more than ASN.1. It specifies both the description format and the encoding. PB is ready to be used out of the box. You have a compact IDL and a performant encoding format without having to think about anything. You have to remember that PB was designed for internal Google use as a tool to solve their problems, not as a generic solution.

ASN.1 is extremely unwieldy in comparaison. It has accumulated a lot of cruft through the year. Plus they don’t provide a default implementation.

Re: Protobuffers Are Wrong (2018)

#290

Post by instig007 »

kentonv wrote: post #215
instig007 wrote: post #192

Earlier quoted context omitted.

> is NOT the same scenario where you get to design your types once and then EVERYTHING that ever touches those types is forced through a type checker. the author never claimed the types had to be designed only once, he claimed that schema evolution chosen by protobuf is inadequate for the purpose of lossless evolution. > Kenton has already provided a good explanation here: https://news.ycombinator.com/item?id=4514059…

> type algebra either doesn't exist or impractical because only PL theorists know about it, not Kenton. Hi I'm Kenton. I, too, was enamored with advanced PL theory in college. Designed and implemented my own purely-functional programming language. Still wish someone would figure out a working version of dependent types for real-world use, mainly so we could prove array bounds-safety without runtime checks. In two dec…

> Still wish someone would figure out a working version of dependent types for real-world use, mainly so we could prove array bounds-safety without runtime checks.

Hi Kenton, I'm not sure what kind of PL theory you studied in college, but "array bounds-safety without runtime checks" don't require dependent types. They are being proven with several available SMT solvers as of right now, just ask LLVM folks with their "LLVM_ENABLE_Z3_SOLVER" compiler flag, the one that people build their real-world solutions on.

By the way, you don't have to say "real-world" in every comment to appeal to your google years as a token of "real-world vs the rest of you". "But my team at google wouldn't use it", or something along that line, right?

https://ats-lang.sourceforge.net/DOCUMENT/INT2PROGINATS/HTML...

Post reply on HN