Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

211–220 of 321 posts

Re: Protobuffers Are Wrong (2018)

#211

Earlier quoted context omitted.

Proto2 let you do this and the "required" keyword was removed because of the problems it introduces when evolving the schema in a system with many users that you don't necessarily control. Let's say you want to add a new required field, if your system receives messages from clients some clients may be sending you old data without the field and now the parse step fails because it detects a missing field. If you ever w…

> Let's say you want to add a new required field, if your system receives messages from clients some clients may be sending you old data without the field and now the parse step fails because it detects a missing field. A parser has to (inherently) neither fail (compatibility mode) nor lose the new field (a passthrough mode), nor allow diverging (strict mode). The fact that capnproto/parser authors don't realize that…

Hi, I'm the apparently-FUD-spreading Cap'n Proto author.

Sure! You could certainly imagine extending Protobuf or Cap'n Proto with a way to specify validation that only happens when you explicitly request it. You'd then have separate functions to parse vs. to validate a message, and then you can perform strict validation at the endpoints but skip it in middleware.

This is a perfectly valid feature idea which many people have entertained an even implemented successfully. But I tend to think it's not worth trying to do have this in the schema language because in order to support every kind of validation you might want, you end up needing a complete programming language. Plus different components might have different requirements and therefore need different validation (e.g. middleware vs. endpoints). In the end I think it is better to write any validation functions in your actual programming language. But I can certainly see where people might disagree.

Re: Protobuffers Are Wrong (2018)

#212
post #200

But why do you need serialization? Because the data structure on disk is not the same as in memory. Arthur Whitney's k/q/kdb+ solved this problem by making them the same. An array has the same format in memory and on disk, so there is no serialization, and even better, you can mmap files into memory, so you don't need cache! He also removed the capability to define a structure, and force you to use dictionary(structu…

> But why do you need serialization? Because the data structure on disk is not the same as in memory.

Not always - in browser applications for example, there is no way to directly access the disk, nevermind mmap().

Re: Protobuffers Are Wrong (2018)

#213

https://news.ycombinator.com/item?id=18190005 Just FYI: an obligatory comment from the protobuf v2 designer. Yeah, protobuf has lots of design mistakes but this article is written by someone who does not understand the problem space. Most of the complexity of serialization comes from implementation compatibility between different timepoints. This significantly limits design space.

Relatedly, most of the author's concerns are solved by wrapping things in a message. > oneof fields can’t be repeated. Wrap oneof field in message which can be repeated > map fields cannot be repeated. Wrap in message which can contain repeated fields > map values cannot be other maps. Wrap map in message which can be a value Perhaps this is slightly inconvenient/un-ergonomic, but the author is positioning these thin…

To clarify. Protobuf’s simplest change is adding a field to a message so wrapping maps of maps, maps of fields, oneof fields into a message makes these play to its strengths. It feels like over engineering to turn your Inventory map of items into a Inventory message, but you will be grateful for it when you need a capacity field later.

Re: Protobuffers Are Wrong (2018)

#214
post #200

But why do you need serialization? Because the data structure on disk is not the same as in memory. Arthur Whitney's k/q/kdb+ solved this problem by making them the same. An array has the same format in memory and on disk, so there is no serialization, and even better, you can mmap files into memory, so you don't need cache! He also removed the capability to define a structure, and force you to use dictionary(structu…

Forget on-disk. Different CPUs represent basic data types with different in-memory representations (endianness). Furthermore different CPUs have different capabilities with respect to how data must be aligned in memory in order to read or write it (aligned/unaligned access). At least historically unaligned access could fault your process. Then there's the problem, that you allude to, that different programming languages use different data layouts (or often a non-standardised layout). If you want communication within a system comprising heterogeneous CPUs and/or languages, you need to translate or standardise your a wire format and/or provide a translation layer aka serialisation.

Re: Protobuffers Are Wrong (2018)

#215

Earlier quoted context omitted.

the complaints about the Protobuf type system being not flexible enough are also really funny to read. fundamentally, the author refuses to contend with the fact that the context in which Protobufs are used -- millions of messages strewn around random databases and files, read and written by software using different versions of libraries -- is NOT the same scenario where you get to design your types once and then EVE…

> is NOT the same scenario where you get to design your types once and then EVERYTHING that ever touches those types is forced through a type checker. the author never claimed the types had to be designed only once, he claimed that schema evolution chosen by protobuf is inadequate for the purpose of lossless evolution. > Kenton has already provided a good explanation here: https://news.ycombinator.com/item?id=4514059…

> type algebra either doesn't exist or impractical because only PL theorists know about it, not Kenton.

Hi I'm Kenton. I, too, was enamored with advanced PL theory in college. Designed and implemented my own purely-functional programming language. Still wish someone would figure out a working version of dependent types for real-world use, mainly so we could prove array bounds-safety without runtime checks.

In two decades building real-world complex systems, though, I've found that getting PL theory right is rarely the highest-leverage way to address the real problems of software engineering.

Re: Protobuffers Are Wrong (2018)

#217

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

> Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used.

What I dislike the most about blog posts like this is that, although the blogger is very opinionated and critical of many things, the post dates back to 2018, protobuf is still dominant, and apparently during all these years the blogger failed to put together something that they felt was a better way to solve the problem. I mean, it's perfectly fine if they feel strongly about a topic. However, investing so much energy to criticize and even throw personal attacks on whoever contributed to the project feels pointless and an exercise in self promotion at the expense of shit-talking. Either you put something together that you feel implements your vision and rights some wrongs, or don't go out of your day to put down people. Not cool.

Re: Protobuffers Are Wrong (2018)

#218
post #108
post #15

I share the author's sentiment. I hate these things. True story: trying to reverse engineer macOS Photos.app sqlite database format to extract human-readable location data from an image. I eventually figured it out, but it was: A base64 encoded Binary Plist format with one field containing a ProtoBuffer which contained another protobuffer which contained a unicode string which contained improperly encoded data (for e…

> This could have been a simple JSON string. There's nothing "simple" about parsing JSON as a serialization format.

Having attempted writing a JSON parser from scratch and a protobuf parser from scratch and only completing one of them, I disagree.

Re: Protobuffers Are Wrong (2018)

#219
post #8

Earlier quoted context omitted.

The best way to get your point across is by starting with ad-hominem attacks to assert your superior intelligence.

Yeah, let's pretend that type algebra doesn't exist, and even if it does exist then it's not useful and definitely isn't practical in data protocols. Let's believe that the authors of protobuf considered everything, and since they aren't amateurs (by the virtue of having worked on protobuf at Google, presumably), every elaborated opinion that draws them as amateurs at applying type algebra in data protocol designs is…

They're not amateurs by virtue of being some of the most senior engineers ever to work at Google. You don't get to play the "ad hominem" card while calling them names. This whole thread is embarrassing.

Re: Protobuffers Are Wrong (2018)

#220

> Protobuffers correspond to the data you want to send over the wire, which is often related but not identical to the actual data the application would like to work with This sums up a lot of the issues I’ve seen with protobuf as well. It’s not an expressive enough language to be the core data model, yet people use it that way. In general, if you don’t have extreme network needs, then protobuf seems to cause more har…

Protobuf is independent from REST. You can have either one. Or both. Or neither. One has nothing to do with the other.

yes I fully understand that, the point is a lot of teams focus on protobuf for their network layer
Post reply on HN