Data serialization

I think it's point is that JSON isn't a great streaming format; which it isn't.

That's not to say there aren't streaming decoders available or workarounds to allow non-streaming decoders to emulate streaming behaviour. However generally JSON parsers require the full JSON object before they can decode.

CSV standards are flawed and it has a lot of weird edge cases.

Moreover, byte encoding formats are more concise, so faster to transport over the wire and store and the schema seldom proves to be an hindrance, as it assists in data validation and acts as documentation.

That's the case with Perl at least. Both JOSN::XS and Cpanel::JSON::XS are faster than Data::MessagePack and Sereal on an iMac 2009 with Yosemite. On Linux Sereal is 53% faster than Data::MessagePack, but the rest of the benchmark is quite similar. CBOR::XS beats just about any data serializer. Sereal is quite sophisticated, it can compress data using snappy, zlib and zstd.

            Rate msgpack  sereal cjsonxs  jsonxs  cborxs
    msgpack 204848/s      --     -2%     -6%    -18%    -34%
    sereal  208207/s      2%      --     -4%    -16%    -33%
    cjsonxs 217825/s      6%      5%      --    -13%    -29%
    jsonxs  249011/s     22%     20%     14%      --    -19%
    cborxs  308571/s     51%     48%     42%     24%      --

All the module versions tested are current. Here is the code for this benchmark:

https://hastebin.com/wazuqovexo.pl

Protobuf has supported this for as long as I’ve used it. You add an optional field for the new data. And there’s a couple other considerations. (Docs: [0])

Fields have a number, which you can/should mark as reserved after you take it out. Then future developers won’t use a field that was something else in the past. [1]

Since version all fields are optional ( they removed the required field feature). So your app has to check that the fields you want (in that version of your app) are present.

[0] https://developers.google.com/protocol-buffers/docs/proto3#u...

"It might be considered out of fashion today in our staged this-vs-that culture, but I don't see anything wrong with it.... no API required."

Well, it has no type system, not even "string vs. number", no hierarchy of any kind (no objects, no lists, nothing going deeper), and CSV/TSV is actually a meta-specification requiring a correct use of CSV to provide a complete specification (since you can trust literally nothing about a CSV/TSV implementation otherwise) so you can handle the distinguished characters in the data values. Whatever it is you are about to say is the answer to that problem is not; it is an answer and there are others in common use, and correct usage needs to specify them.

Unlike the formats listed, it's completely unclear how to serialize arbitrary objects into it in a way that other people would agree to sight-unseen. That is, yes, I'm aware you can jam anything you want into CSV. For instance, you could make one of the columns be JSON. But that defeats the purpose of claiming it's good enough on its own, no? Same for any other scheme you might propose; unless you can show it to be a standard, it's not an answer, just an ad-hoc solution.

I'm not saying it's useless. Literally used it last Friday to dump some data into something that could be loaded by a spreadsheet program so someone can mix & match the data without having to ask for every individual possible view to come straight from me. Very useful format at times. But as a generalized serialization format? It's not even in the running, which is precisely why it doesn't come up in these discussions.

If you want to get concrete, show me the canonical way you'd lossly serialize the DOM tree nodes for the HN home page into CSV, in a way that you are reasonably sure is the same way everybody else would tend to if given the task, then show me the emitter and generator code that is more convenient than the alternatives.