Don't Use Protobuf for Telemetry

I think that's one reason, but there are others too, like getting generated type-safe accessors and an explicit schema. I've sometimes used protobuf for this reason even when I'm not planning to use binary format.

> I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

Here is a breakdown of a binary that uses upb and links in binary format, JSON format, text format, and some generated protos:

         VM SIZE
     ---------------
      20.6%  26.5Ki    Generated Reflection
        45.7%  12.1Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upbdefs.c
        39.3%  10.4Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upbdefs.c
         5.0%  1.33Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upbdefs.c
         3.1%     832    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upbdefs.c
         2.4%     648    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upbdefs.c
         1.2%     320    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/timestamp.upbdefs.c
         1.2%     312    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upbdefs.c
         1.1%     304    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/field_mask.upbdefs.c
         1.1%     296    [1 Others]
      19.1%  24.5Ki    JSON
        60.4%  14.8Ki    upb/json_decode.c
        39.6%  9.67Ki    upb/json_encode.c
      14.7%  18.9Ki    Reflection Runtime
        81.9%  15.4Ki    upb/def.c
        18.1%  3.41Ki    upb/reflection.c
      13.9%  17.8Ki    Generated Code
        46.9%  8.34Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upb.c
        17.6%  3.12Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upb.c
        16.4%  2.92Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upb.c
        14.7%  2.61Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/descriptor.upb.c
         2.0%     356    bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upb.c
         1.4%     264    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upb.c
         0.5%      88    [2 Others]
         0.3%      48    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/any.upb.c
         0.3%      48    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upb.c
      10.0%  12.9Ki    Binary Format
        54.0%  6.95Ki    upb/decode.c
        46.0%  5.92Ki    upb/encode.c
       8.8%  11.3Ki    Core Runtime
        66.3%  7.52Ki    upb/table.c
        17.8%  2.02Ki    upb/upb.c
        15.9%  1.80Ki    upb/msg.c
       5.6%  7.19Ki    [34 Others]
       3.8%  4.94Ki    Text Format
       100.0%  4.94Ki    upb/text_encode.c
       3.5%  4.46Ki    [section .rodata]
     100.0%   128Ki    TOTAL

But it is all a la carte: if you don't want JSON, just don't use it and don't link it in.

(btw that 7Ki for upb/table.c is to support protobuf maps, I want to make that a la carte also so that you don't pay for it when you don't use maps).

Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. They just did something that worked well enough, and it worked well enough that it was rapidly adopted by the rest of the company. It was then too late to change anything.

Yes, the fact that a variable-width size must be written before the data is a kind of big problem, which essentially requires you to make two passes over the message tree, one to compute sizes and one to write the data. There are some clever optimizations that can reduce the impact, but I don't think the designers would make the same decision if starting from scratch with no need to support legacy. But there was no point in history where it was worth breaking compatibility to fix this issue, so that's how it remains. It's a problem, just not that big a problem.

(I maintained Protobuf for several years, including writing version 2 and open sourcing it.)

That's a bit harsh. Protobufs deliver smaller wire size than any of the newer "zero-copy" formats. And many receivers of zero-copy formats will... copy the data into some internal representation. If your protobuf implementation delivers classes that are good enough to work with internally (store in maps, forward, etc) then you don't really lose something; instead you gain, due to no manual conversion layer.

That's not my position at all. In my other comment (https://news.ycombinator.com/item?id=25586447) I explain how I've spent 10 years trying to improve on protobuf C++ precisely because I agree that some of these limitations are unnecessary.

> No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications.

I suppose it depends what you are comparing it to. Almost every JSON library has the same limitations you mentioned, and yet many people find JSON useful for network applications. But I agree that giving users full control over allocations makes a library useful in many more situations.

I think arena allocation is a pretty reasonable solution to the problem. You can use whatever memory you want for the arena (stack, heap, static buffer) and you can constrain it so that no heap allocations are allowed.

Unfortunately protobuf C++ can't live fully within this arena model while it uses std::string for accessors. Hopefully this can be fixed at some point.