Arguing against using protobuffers

Good to know about the 0x00 trick.

My point about metadata is that if protobufs had even a small amount of self description in them, the middlemen who weren't being updated could all have been found automatically and the version skew issue would have been much less of an issue. Like how Dapper can follow RPCs around, but for data and running binaries.

Google doesn't do that because for its specific domain it needs a very tight encoding, amongst other reasons (and the legacy issues). It could have fixed the validating-but-not-updated-middleman issue in other ways, but instead it made the schema type system less rigorous vs more rigorous. That seems the wrong direction.

Row polymorphism is a more principled approach that makes the type theorists happy.

The point of row polymorphism is not just that you can say "I may or may not have other fields, but I definitely have a 'name' and 'email' field", but that the extra unknown fields have a name that can be used to constrain other types.

For example, you can have this function type:

{name: string, email: string, &rest} -> {name: string, email: string, &rest}

This is different than this function type:

{name: string, email: string, &rest} -> {name: string, email: string}

The first type can act as a forward compatible passthrough because it says that whatever extra fields are in the input are also in the output. The second type promises that its output only has name and email fields.

The same applies to variant types: you can say that a variant has option A, option B, and other options rest:

(A | B | &rest) -> (A | B | &rest)

vs

(A | B | &rest) -> (A | B)

Functions of these types are polymorphic in the schema. For instance, the type (A | B | &rest) -> (A | B) can be instantiated with rest = (C | D) to get (A | B | C | D) -> (A | B).

So row types are fundamentally different in that it's the functions that that explicitly deal with multiple possible schemas. At the end of the day all of the rest variables get instantiated with explicit types to get a concrete instantiation of a function in which all the rest variables have been replaced by concrete types.

A serialisation library could use this to do version negotiation automatically. After it has negotiated a version, the library instantiates the functions so that the rest type variables get instantiated with the actual concrete version of the schema.

Different languages implement row types in different ways. Some compile row types in a way akin to Java's type erasure. They compile a single version of each polymorphic function that can be used regardless of how the rest parameters get instantiated. Some compile row types in a way akin to C#'s generics or C++ templates: they compile a separate version for each instantiation.

The advantage of the latter is that the data representation can be optimised with full knowledge of the concrete schema. If we have a function of type {name: string, email: string, &rest} -> {name: string, email: string, &rest} instantiated with rest = {age: int} then that compiles to a version of type {name: string, email: string, age: int} -> {name: string, email: string, age: int}. This compiles to faster code because the compiler statically knows the size of the thing.

In a client-server situation you wouldn't know the schema of rest until run time, so you'd need either have a JIT compiler that can compile new versions at run time, or specify a fixed number of options for rest at compile time. To update a client-server application you'd need to recompile both the client and server with support for the new version. That's not nice but it does not have a chicken and egg problem because both the client and server still support the old version too.

TL;DR: with row types schemas are always rigidly defined, it's the functions that can handle multiple schemas.

My C code didn't include mention of ints though, so I'm wondering where you got that from.

Your first example is UB and again, is not something my example depended on.

Your final claims are overly cautious. It is perfectly fine to use uint32_t in this way. Uint32_t is defined as a 32-bit unsigned integer. There is a bijection between network order 32-bit unsigned integers and host order integers, and ntohs is the bijection. It is no different than storing any other value. It is certainly not wrong.

> You are missing when you have a middle layer.

And storage. People might have petabytes of historic data stored in protobufs.

> Suppose I'm an application author and I have an integer field called "quantity" which decoded to a 0. How can I tell whether that 0 meant "the quantity was 0 in the database" or "the quantity field was missing" instead?

First, this is clear on the level of wire encoding: either the field has encoded 0 value, or it is simply missing from encoding.

Second, in proto2, you actually have has_quantity() method on a proto message, which will tell you whether quantity is missing or set to 0.

In proto3, the design decision was that the has_foo() methods are available only on embedded message field, and not available on primitive fields, so you'd have to wrap your int64 in a message wrapper, like e.g. the ones available in google/protobuf/wrappers.proto.

The point here (and a common pattern inside google3) is that in your handling code you simply manually check the presence of all required fields: if (!foo.has_quantity()) { return FailedPreconditionError("missing quantity"); }. It is a bit of a hassle, but the benefit is that you have control on where the bug originates and how it is handled in your application layer, as opposed to silently dropping the whole proto message on the floor.