Live data from Hacker News

From XML to JSON to CBOR

cborbook.com

91–100 of 106 posts

Re: From XML to JSON to CBOR

#91
post #41

Oh good, another CBOR thread. Disclaimer: I wrote and maintain a MessagePack implementation. I've also bird dogged this for a while, HN search me. Mostly, I just want to offer a gentle critique of this book's comparison with MessagePack [0]. > Encoding Details: CBOR supports indefinite-length arrays and maps (beneficial for streaming when total size is unknown), while MessagePack typically requires fixed collection c…

> Well, CBOR is MessagePack. Carsten Bormann forked MessagePack

Sure, that’s sort of true but missing context. Bormann (and others) wanted to add things such as separate string and byte sequence types. The MessagePack creator refused for years. Fair enough it’s his format. But it frustrated the community dealing with string vs bytes issues. It also highlights a core philosophical difference of a mostly closed spec vs an extensible first one.

> changed some of the tag values, wrote a standard around it, and submitted it to the IETF against the wishes of MessagePack's creators.

That’s just incorrect and a childish way to view it in my opinion.

The core philosophy and mental models are different in key aspects.

MessagePack is designed as a small self mostly closed format. It uses a simple TLV format with a couple hundred possible user extensions and some clever optimizations. The MP “spec” focuses on this.

CBOR re-envisioned the core idea of MessagePack from the ground up as an extensible major/minor tag system. It’s debatable how much CBOR is a fork of MPack vs a new format with similarities.

The resulting binary output is pretty similar with similar benefits but the core theoretical models are pretty different. The IETF standard bares little to no resemblance to the MessagePack specification.

> The facilities are the same (well, the tag is 8 bytes instead of 1 byte, but w/e); it's TLV all the way down (Bormann ripped this also).

The whole point of CBOR is that the tags go from 1-8 bytes. The parser designs end up fairly different due to the different tag formats. I’ve written and ported parsers for both.

It’s not like the MessagePack creator invented TLV formats either. He just created an efficient and elegant one that’s pretty general. No one says he ripped off “TLV”.

You can’t just take a message pack parser and turn it into a CBOR one by changing some values. I’ve tried and it turns out poorly and doesn't support much of CBOR.

> This refers to CBOR's indefinite length types, but awkwardly, streaming is a protocol level feature, not a data format level feature.

The indefinite length format is very useful for embedded space. I’ve hit limits with MessagePack before on embedded projects because you need to know the length of an array upfront. I wished I’d had CBOR instead.

This can also be useful for data processing applications. For example streaming the conversion of a large XML file into a more concise CBOR format would be much more memory efficient. For large scale that’s pretty handy.

> > However, MessagePack sacrifices human-readability > This, of course, applies to CBOR as well.

For the binary format yes. However the CBOR specification defines an official human readable text format for debugging and documentation purposes. It also defines a schema system like json-schema but for CBOR.

Turns out “just some specs” can actually be pretty valuable.

Re: From XML to JSON to CBOR

#92

Fun fact: CBOR is used within the WebAuthn (Passkey) protocol. To do Passkey-verification server-side, I had to implement a pure-SQL/PLpgSQL CBOR parser, out of fear that a C-implementation could crash the PostgreSQL server: https://github.com/truthly/pg-cbor

And .Net 5 circa 2020 added support for CBOR. ASP.NET ended up being a good choice for an experimental WebAuthn server for FedCM and DID experiments.

Re: From XML to JSON to CBOR

#93
post #69

Fun fact: CBOR is used within the WebAuthn (Passkey) protocol. To do Passkey-verification server-side, I had to implement a pure-SQL/PLpgSQL CBOR parser, out of fear that a C-implementation could crash the PostgreSQL server: https://github.com/truthly/pg-cbor

That’s why I’m wondering if there is an actual CBOR encoder in the browsers? I mean, there must be one, or am I wrong?

Yes.[1][2]

[1] https://source.chromium.org/chromium/chromium/src/+/main:com...

[2] https://source.chromium.org/chromium/chromium/src/+/main:dev...

Re: From XML to JSON to CBOR

#94
post #41

Oh good, another CBOR thread. Disclaimer: I wrote and maintain a MessagePack implementation. I've also bird dogged this for a while, HN search me. Mostly, I just want to offer a gentle critique of this book's comparison with MessagePack [0]. > Encoding Details: CBOR supports indefinite-length arrays and maps (beneficial for streaming when total size is unknown), while MessagePack typically requires fixed collection c…

> Well, CBOR is MessagePack. Carsten Bormann forked MessagePack Sure, that’s sort of true but missing context. Bormann (and others) wanted to add things such as separate string and byte sequence types. The MessagePack creator refused for years. Fair enough it’s his format. But it frustrated the community dealing with string vs bytes issues. It also highlights a core philosophical difference of a mostly closed spec vs…

I am really glad you replied.

> Sure, that’s sort of true but missing context. Bormann (and others) wanted to add things such as separate string and byte sequence types. The MessagePack creator refused for years. Fair enough it’s his format. But it frustrated the community dealing with string vs bytes issues.

msgpack-ruby added string support less than a month after cbor-ruby's first commit [0] [1]. The spec was updated over two months before [2]. Awful lot of work if this were really just about strings.

> It also highlights a core philosophical difference of a mostly closed spec vs an extensible first one.

MP has been always been extensible, via ext types.

> That’s just incorrect

I am entirely correct [3].

> MessagePack is designed as a small self mostly closed format.

Isn't it a lot of effort to get an IETF standard changed? Isn't that the benefit of a standard? You keep saying "mostly closed" like it's bad. Data format standards in particular really shouldn't change: who knows how many zettagottabytes there are stored in previous versions?

> It’s debatable how much CBOR is a fork of MPack vs a new format with similarities.

cbor-ruby is literally a fork of msgpack-ruby. The initial commit [0] contains headers like:

    /\*
     \* CBOR for Ruby
     \*
     \* Copyright (C) 2013 Carsten Bormann
     \*
     \*    Licensed under the Apache License, Version 2.0 (the "License").
     \*
     \* Based on:
     \*\*\*\*\**/
    /*
     \* MessagePack for Ruby
     \*
     \* Copyright (C) 2008-2013 Sadayuki Furuhashi
> The resulting binary output is pretty similar with similar benefits

This is the whole game isn't it? The binary output is pretty similar? These are binary output formats!

> but the core theoretical models are pretty different.

I think you're giving a little too much credence to the "theoretical model". It's not more elegant to do what cbor-ruby does [4] vs. what MP does [5] (this is my lib). I literally just use the tag value, or for fixed values I OR them together. The format is designed for you to do this. What's more elegant than a simple, predefined value?

> The whole point of CBOR is that the tags go from 1-8 bytes.

The tags themselves are only 1 byte, until you get to extension types.

> The parser designs end up fairly different due to the different tag formats.

The creator of CBOR disagrees: cbor-ruby was a fork of msgpack-ruby with the tag values changed.

> No one says he ripped off “TLV”.

Don't conflate the general approach with literally forking an existing project.

> You can’t just take a message pack parser and turn it into a CBOR one by changing some values.

This is a strawman. My claim has been about the origins of CBOR, not how one can transmute an MP codec to a CBOR codec.

> I’ve hit limits with MessagePack before on embedded projects because you need to know the length of an array upfront.

When everything's fine, sure this works. If there are any problems whatsoever, you're totally screwed. Any protocol that supports streaming handles this kind of thing. CBOR doesn't. That's bad!

> For example streaming the conversion of a large XML file into a more concise CBOR format would be much more memory efficient.

It's probably faster to feed it through zstd. Also I think you underestimate how involved it'd be to round-trip a rich XML document to/from CBOR/MP.

> However the CBOR specification defines an official human readable text format for debugging and documentation purposes.

Where? Are you talking about Diagnostic Notation [6]? Hmm:

"Note that this truly is a diagnostic format; it is not meant to be parsed. Therefore, no formal definition (as in ABNF) is given in this document. (Implementers looking for a text-based format for representing CBOR data items in configuration files may also want to consider YAML [YAML].)"

YAML!? Anyway, it literally doesn't define it.

[0]: https://github.com/msgpack/msgpack-ruby/commit/60e846aaaa638...

[1]: https://github.com/cabo/cbor-ruby/commit/5aebd764c3a92d40592...

[2]: https://github.com/msgpack/msgpack/commit/5dde8c4fd0010e1435...

[3]: https://github.com/msgpack/msgpack/issues/129#issuecomment-1...

[4]: https://github.com/cabo/cbor-ruby/blob/5aebd764c3a92d4059236...

[5]: https://github.com/camgunz/cmp/blob/master/cmp.c#L30

[6]: https://www.rfc-editor.org/rfc/rfc8949.html#name-diagnostic-...

Re: From XML to JSON to CBOR

#95
I prefer DER, which is also a binary format so it has the advantages of binary formats, too. (There is also BER, but in my opinion, DER is better.) I use DER in some programs, if the structured data format is useful. (Also, since text format is sometimes useful too, I had made up TER which is intended to be converted to DER. The DER file can be made in other ways as well and it is not required to use TER.)

(Also, standard ASN.1 does not have a key/value list type (which JSON and CBOR do have), but I had made up some nonstandard extensions to ASN.1 (called ASN.1X), including a few additional types, one of which is the key/value list type. Due to this, ASN.1X can now make a superset of the data that can be made by JSON (the only new type that is needed for this is the key/value list type; the other types of JSON are already standard ASN.1 types).)

Re: From XML to JSON to CBOR

#96
post #75
post #29

ASN.1 while complex has really seems to be a step up from those (even if older) in terms of terseness (as binary encoding) and generality.

Would you rather write a parser for this: SEQUENCE { SEQUENCE { OBJECT IDENTIFIER '1 2 840 113549 1 1 1' NULL } BIT STRING 0 unused bits, encapsulates { SEQUENCE { INTEGER 00 EB 11 E7 B4 46 2E 09 BB 3F 90 7E 25 98 BA 2F C4 F5 41 92 5D AB BF D8 FF 0B 8E 74 C3 F1 5E 14 9E 7F B6 14 06 55 18 4D E4 2F 6D DB CD EA 14 2D 8B F8 3D E9 5E 07 78 1F 98 98 83 24 E2 94 DC DB 39 2F 82 89 01 45 07 8C 5C 03 79 BB 74 34 FF AC 04 AD 15…

That is a text format, although DER is a binary format and encodes the data which there is represented by text. I think they should not have made a bit string (or octet string) to encapsulate another ASN.1 data and would be better to put it directly, but nevertheless it can work. The actual data to be parsed will be binary, not the text format like that.

DER is a more restricted variant of BER and I think DER is better than BER. PEM is also DER format but is encoded as base64 and has a header to indicate what type of data is being stored, rather than directly.

Re: From XML to JSON to CBOR

#97
post #73
post #29

ASN.1 while complex has really seems to be a step up from those (even if older) in terms of terseness (as binary encoding) and generality.

The FOSS tooling for it sucks balls. That's why

Then, work to make a better one. (I had written a C library to read/write DER format, although it does not deal with the schema.)

Re: From XML to JSON to CBOR

#98

Erlang / Elixir has amazing support for ASN.1! I love it. https://www.erlang.org/doc/apps/asn1/asn1_getting_started.ht... https://www2.erlang.org/documentation/doc-14/lib/asn1-5.1/do... ( https://www2.erlang.org/documentation/doc-14/lib/asn1-5.1/do... ) I am using ASN.1 to communicate between a client (Java / Kotlin) and server (Erlang / Elixir), but unfortunately Java / Kotlin has somewhat of a shitty support for AS…

I also use ASN.1 but I use C, so I wrote my own implementation of DER.

Re: From XML to JSON to CBOR

#99

Love or hate JSON, the beauty and utility stem from the fact that you have only the fundamental datatypes as a requirement, and that's it. Structured data that, by nesting, pleases the human eye, reduced to the max in a key-value fashion, pure minimalism. And while you have to write type converters all the time for datetime, BLOBs etc., these converters are the real reasons why JSON is so useful: every OS or framewor…

> you have only the fundamental datatypes as a requirement

Not really; the set of datatypes has problems. It uses Unicode, not binary data and not non-Unicode text. Numbers are usually interpreted as floating point numbers rather than integers, which can also be a problem. Keys can only be strings. And, other problems. So, the data types are not very good.

And, since it is a text format, it means that escaping is required.

> And while you have to write type converters all the time for datetime, BLOBs etc.

Not having a proper data type for binary means that you will need to encode it using different types and then avoids the benefit of JSON, anyways. So, I think JSON is not as helpful.

I think DER is better (you do not have to use all of the types; only the types that you are using is necessary to be implemented, because the format of DER makes it possible to skip anything that you do not care about), and I made up TER which is text based format which can be converted to DER (so, even though a binary data is represented as text, it is still representing the binary data type, rather than needing to use the wrong data type like JSON does).

> And you can complain or explain with JSON: "Comments not a feature?! WTF!" - Add a field with the key "comment"

But then it is a part of the data, which you might not want.

Re: From XML to JSON to CBOR

#100
post #41

Oh good, another CBOR thread. Disclaimer: I wrote and maintain a MessagePack implementation. I've also bird dogged this for a while, HN search me. Mostly, I just want to offer a gentle critique of this book's comparison with MessagePack [0]. > Encoding Details: CBOR supports indefinite-length arrays and maps (beneficial for streaming when total size is unknown), while MessagePack typically requires fixed collection c…

> This refers to CBOR's indefinite length types, but awkwardly, streaming is a protocol level feature, not a data format level feature.

BER also has indefinite length as well as definite length, but the way that it is doing, is not very good (DER only uses definite length). I think it is more helpful to use a different format when streaming with indefinite length is require, so I made up DSER (and SDSER) which is working as follows:

- The type, which is encoded same as DER.

- If it is constructed, all items it contains come next (the length is omitted).

- If it is primitive, zero or more segments, each of which starts with one byte in range 0x01 to 0xFF telling how many bytes of data are in that segment. (The value is then just the concatenation of all segments together.)

- For both primitive and constructed, one byte with value 0x00 is the termination code.

> Bormann's contribution is the registry, which is bonkers [1]. There's... dozens of extensions there? Hundreds? No CBOR implementation supports anywhere near all this stuff.

It should not need to support all of that stuff; you will only use the ones that are relevant for your program. (There is also the similar kind of complaint with ASN.1, and the similar response that I had made.)

> If something is in high demand, but doesn't have good support across platforms, then you're putting extra burden on those platforms. Ex: it's not great if my tiny microcontroller now has to support bignums or 128-bit UUIDs.

Although it is a valid concern, you would use data which does not have numbers bigger than you need to be, so it can avoid such a problem. You can treat UUIDs like octet strings, although if you only need small numbers then you should use the small numbers types instead, anyways.

> If something isn't in high demand or can't easily be supported across platforms, but you want support for it anyway, there's no need to tell anyone else you're using that thing.

Sometimes it is useful to tell someone else that you are using that thing, although often it is unnecessary, like you said.

Post reply on HN