Live data from Hacker News

From XML to JSON to CBOR

cborbook.com

71–80 of 106 posts

Re: From XML to JSON to CBOR

#71
post #59

Earlier quoted context omitted.

Notably missing is a comparison to Cap'n Proto, which to me feels like the best set of tradeoffs for more binary interchange needs. I honestly wonder sometimes if it's held back by the name— I love the campiness of it, but I feel like it could be a barrier to being taken seriously in some environments.

Doesn't Cap'n Proto require the receiver to know the types for proper decoding? This wouldn't entirely disqualify it from comparison, since e.g. protobufs are that way as well, but they make it less interesting for comparing to CBOR, which is type-tagged.

There's quite a few formats that are self-describing already, so having a format that can skip the type and key tagging for that extra little bit of compactness and decoding efficiency is a unique selling point.

There's also nothing stopping you from serializing unstructured data using an array of key/value structs, with a union for the value to allow for different value types (int/float/string/object/etc), although it probably wouldn't be as efficient as something like CBOR for that purpose. It could make sense if most of the data is well-defined but you want to add additional properties/metadata.

Many languages take unstructured data like JSON and parse them into a strongly-typed class (throwing validation errors if it doesn't map correctly) anyways, so having a predefined schema is not entirely a bad thing. It does make you think a bit harder about backwards-compatibility and versioning. It also probably works better when you own the code for both the sender and receiver, rather than for a format that anyone can use.

Finally, maybe not a practical thing and something that I've never seen used in practice: in theory you could send a copy of the schema definition as a preamble to the data. If you're sending 10000 records and they all have the same fields in the same order, why waste bits/bytes tagging the key name and type for every record, when you could send a header describing the struct layout. Or if it's a large schema, you could request it from the server on demand, using an id/version/hash to check if you already have it.

In practice though, 1) you probably need to map the unknown/foreign schema into your own objects anyways, and 2) most people would just zlib compress the stream to get rid of repeated key names and call it a day. But the optimizer in me says why burn all those CPU cycles decompressing and decoding the same field names over and over. CBOR could have easily added optional support for a dictionary of key strings to the header, for applications where the keys are known ahead of time, for example. (My guess is that they didn't because it would be harder for extremely-resource-constrained microcontrollers to implement).

Re: From XML to JSON to CBOR

#72
post #70
post #35

Earlier quoted context omitted.

I think parsing BSON is simpler than parsing JSON, BSON has additional types but the top level is always a document. Whereas the following are all valid JSON: - `null` - `"hello"` - `[1,2,NaN]` Additionally, BSON will just tell you what the type of a field is. JSON requires inferring it.

NaN is not part of JSON by any spec. Top level scalar values were disallowed by RFC 4627.

Fair enough. I'm not sure how much JSON parsers in the wild care about that spec. I just tried with Python and it was happy to accept scalars and NaN. JavaScript rejected NaN but was happy to accept a scalar. But sure, compliant parsers can disregard those cases.

Re: From XML to JSON to CBOR

#73
post #29

ASN.1 while complex has really seems to be a step up from those (even if older) in terms of terseness (as binary encoding) and generality.

The FOSS tooling for it sucks balls. That's why

Re: From XML to JSON to CBOR

#74
If GML was an infant, SGML is the bright youngster far exceeds expectations and made its parents too proud, but XML is the drug-addicted gang member who had committed his first murder before he had sex, which was rape.

https://www.schnada.de/grapt/eriknaggum-xmlrant.html

We're going to have to think up something worse for CBOR.

Re: From XML to JSON to CBOR

#75
post #29

ASN.1 while complex has really seems to be a step up from those (even if older) in terms of terseness (as binary encoding) and generality.

Would you rather write a parser for this:

    SEQUENCE {
      SEQUENCE {
        OBJECT IDENTIFIER '1 2 840 113549 1 1 1'
        NULL
        }
      BIT STRING 0 unused bits, encapsulates {
          SEQUENCE {
            INTEGER
              00 EB 11 E7 B4 46 2E 09 BB 3F 90 7E 25 98 BA 2F
              C4 F5 41 92 5D AB BF D8 FF 0B 8E 74 C3 F1 5E 14
              9E 7F B6 14 06 55 18 4D E4 2F 6D DB CD EA 14 2D
              8B F8 3D E9 5E 07 78 1F 98 98 83 24 E2 94 DC DB
              39 2F 82 89 01 45 07 8C 5C 03 79 BB 74 34 FF AC
              04 AD 15 29 E4 C0 4C BD 98 AF F4 B7 6D 3F F1 87
              2F B5 C6 D8 F8 46 47 55 ED F5 71 4E 7E 7A 2D BE
              2E 75 49 F0 BB 12 B8 57 96 F9 3D D3 8A 8F FF 97
              73
            INTEGER 65537
            }
          }
      }
or this:

    (public-key
      (rsa
        (e 65537)
        (n
         165071726774300746220448927123206364028774814791758998398858897954156302007761692873754545479643969345816518330759318956949640997453881810518810470402537189804357876129675511237354284731082047260695951082386841026898616038200651610616199959087780217655249147161066729973643243611871694748249209548180369151859)))
I know that I’d prefer the latter. Yes, we could debate whether the big integer should be a Base64-encoded binary integer or not, but regardless writing a parser for the former is significantly more work.

And let’s not even get started with DER/BER/PEM and all that insanity. Just give me text!

Re: From XML to JSON to CBOR

#76
post #56

Earlier quoted context omitted.

NaN is not a valid value in JSON. Neither are 0123 or .123 (there must always be at least one digit before the decimal marker, but extraneous leading zeroes are disallowed). JSON was originally parsed in javascript with eval() which allowed many things that aren't JSON through, but that doesn't make JSON more complex.

That’s my point, though! I’ve run into popular JSON libraries that will emit all of those! 9007199254740993 is problematic because it’s not representable as a 64 bit float. Python’s JSON library is happy to write it, even though you need an int to represent it, and JSON doesn’t have ints. Edit: I didn’t see my thought all the way through here. Syntax typing invites this kind of nonconformity, because different progra…

> 9007199254740993 is problematic because it’s not representable as a 64 bit float. Python’s JSON library is happy to write it, even though you need an int to represent it

JSON numbers have unlimited range in terms of the format standard, but implementations are explicitly permitted to set limits on the range and precision they generate and handle, and users are warned that:

   [...] Since software that implements IEEE 754 binary64 (double precision)
   numbers is generally available and widely used, good interoperability can be 
   achieved by implementations that expect no more precision or range than these
   provide, in the sense that implementations will approximate JSON
   numbers within the expected precision.
Also, you don't need an int to represent it (a wide enough int will represent it, so will unlimited precision decimals, wide enough binary floats -- of standard formats, IEEE 754 binary128 works -- etc.).

Re: From XML to JSON to CBOR

#77

Erlang / Elixir has amazing support for ASN.1! I love it. https://www.erlang.org/doc/apps/asn1/asn1_getting_started.ht... https://www2.erlang.org/documentation/doc-14/lib/asn1-5.1/do... ( https://www2.erlang.org/documentation/doc-14/lib/asn1-5.1/do... ) I am using ASN.1 to communicate between a client (Java / Kotlin) and server (Erlang / Elixir), but unfortunately Java / Kotlin has somewhat of a shitty support for AS…

Erlang and ASN.1 are from telecom, so it makes sense they have best support

Re: From XML to JSON to CBOR

#78

Earlier quoted context omitted.

CBOR has some stuff that is nice but would be annoying to reimplement. Like using more bytes to store large numbers than small ones. If you need a quick multipurpose binary format, CBOR is pretty good. The only alternative I’d make manually is just memcpy the bytes of a C struct directly to disk and hope that I won’t encounter a system with different endianness.

These days you don't have to worry about endianness much (unless you dealing with raw network packets). However, you do need to worry about byte-padding. Different compilers/systems will place byte padding between items in your struct differently (depending on the contents and ordering of items), and if you are not careful the in-memory or on-disk placement of struct data elements can be misaligned on different syste…

Yeah I try to make sure I do the extern c. I’m also on x86 so I just pretend that alignment is not an issue and I think it works.

Re: From XML to JSON to CBOR

#79
post #75
post #29

ASN.1 while complex has really seems to be a step up from those (even if older) in terms of terseness (as binary encoding) and generality.

Would you rather write a parser for this: SEQUENCE { SEQUENCE { OBJECT IDENTIFIER '1 2 840 113549 1 1 1' NULL } BIT STRING 0 unused bits, encapsulates { SEQUENCE { INTEGER 00 EB 11 E7 B4 46 2E 09 BB 3F 90 7E 25 98 BA 2F C4 F5 41 92 5D AB BF D8 FF 0B 8E 74 C3 F1 5E 14 9E 7F B6 14 06 55 18 4D E4 2F 6D DB CD EA 14 2D 8B F8 3D E9 5E 07 78 1F 98 98 83 24 E2 94 DC DB 39 2F 82 89 01 45 07 8C 5C 03 79 BB 74 34 FF AC 04 AD 15…

The ASN.1 notation wasn't meant for parsing. And then people started writing parsing generators for it, so they adapted. However, you're abusing a text format for human reading and pretending it's a serialization format.

The BER/PER are binary formats and great where binary formats are needed. You also have XER (XML) and JER (JSON) if you want text. You can create an s-expr encoding if you want.

Separate ASN.1--the data model from ASN.1--the abstract syntax notation (what you wrote) from ASN.1's encoding formats.

[1] https://www.itu.int/en/ITU-T/asn1/Pages/asn1_project.aspx

Re: From XML to JSON to CBOR

#80
post #61
post #41

Oh good, another CBOR thread. Disclaimer: I wrote and maintain a MessagePack implementation. I've also bird dogged this for a while, HN search me. Mostly, I just want to offer a gentle critique of this book's comparison with MessagePack [0]. > Encoding Details: CBOR supports indefinite-length arrays and maps (beneficial for streaming when total size is unknown), while MessagePack typically requires fixed collection c…

> ...awkwardly, streaming is a protocol level feature, not a data format level feature. Indeed. I recall that tnetstrings were intentionally made non-streamable to discourage people from trying to do so: "If you need to send 1000 DVDs, don't try to encode them in 1 tnetstring payload, instead send them as a sequence of tnetstrings as payload chunks with checks and headers like most other protocols" > Warning: I have…

> However, a global tag registry can still potentially be useful in that, given CBOR encoded data with a tag that my decoder doesn't support, it may be easier for a human to infer the intended meaning.

Yeah if MP is conservative and CBOR is progressive, I'm slightly less conservative than MP: I'd support UUIDs and bignums. But again, they'd have to be very optional, like in the "we're only reserving these tags, not in any way mandating support" sense.

Post reply on HN