Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I stopped using JSON for my APIs

aloisdeniel.com

Post reply on HN
Print view View original

231–240 of 245 posts

Re: Why I stopped using JSON for my APIs

#231

Post by zzo38computer »

morshu9001 wrote: post #216
cryptonector wrote: post #199

Earlier quoted context omitted.

GeneralString relies on control characters to "load" character sets into the C0 and C1 registers. This is madness -- specifically it's pre-Unicode madness, but before Unicode it made sense.

Oh gosh. Fair enough that this exists and something uses it, but I'd absolutely want to handle that on the ends only, not get asn1 involved in parsing it.

ASN.1 does not necessarily need to get involved in parsing the values; for some applications doing so is unnecessary anyways (this is true for many fields of many types and not only this one, though). ASN.1 will need to be involved in parsing the framing; whether or not it is involved in parsing the values depends on whether the application requires it for that specific value (for example, it is commonly not necessary to parse OIDs (you can usually just treat them as opaque data which can be compared for equality (or looked up in a table), although sometimes it is useful to display them), although some implementations insist on doing so anyways).

Re: Why I stopped using JSON for my APIs

#232

Post by paulddraper »

WorldMaker wrote: post #188
paulddraper wrote: post #120

Earlier quoted context omitted.

BSON is not simply a binary encoding of JSON. It is a JSON superset with binary encoding, created by MongoDB. (And there is even a JSON encoding of BSON, called extended JSON.) AFAIK there is no widely adopted binary pure adaptation of JSON. (There are application-specific storage formats, like PostgreSQL JSONB, or SQLite JSONB.) ——- Moreover, JSON is relatively compact. BSON or other self descriptive binary formats…

CBOR [0] is probably the closest to a widely adopted "pure" adaptation of JSON. It is still technically a superset of JSON, but it tries to be closely matched and frequently cross-references the JSON specs directly, especially because it is also an IETF tracked standard like JSON, and in so far as widely adopted is concerned is included in web standards like WebAuthn today. (For instance, you can't handle Passkeys wi…

> JSON compresses extremely well even in ancient gzip

Correct. MessagePack comes out significantly ahead for small messages, otherwise it's nearly a wash.

> CBOR is probably the closest to a widely adopted "pure" adaptation of JSON

It adds (1) binary data (2) numeric variants including big integers (3) timestamps (4) tagged/semantic types.

I.e. it adds quite a bit. That's not necessarily a bad thing. Browser implementation would be great, but I'm skeptical.

Re: Why I stopped using JSON for my APIs

#233

Post by [deleted] »

ablob wrote: post #230
coolThingsFirst wrote: post #228

Earlier quoted context omitted.

How on earth would gzipping larger amount of data be more efficient than gzipping smaller amount of data?

It's a question of entropy. Data is rarely truly random and for larger data there is a lot higher chance of having this "unrandomness" occur. If your data consists of 4 kilobytes of just 00_01, then you gain a lot by just remembering: "write 00_01 2000 times". Conversely, if the small amount of data is 00_01_00_01_00_01 then using the previous format would yield: "write 00_01 3 times" As you can see, it does not near…

[deleted]

Re: Why I stopped using JSON for my APIs

#234

Post by zzo38computer »

cryptonector wrote: post #215
zzo38computer wrote: post #209

Earlier quoted context omitted.

> Have you written up anything about ASN.1X anywhere? I'd love to take a look. ASN1_BCD_STRING (64): Represents a string with the following characters: "0123456789*#+-. " (excluding the quotation marks). Each octet encodes two characters, where the high nybble corresponds to the first character and the low nybble corresponds to the second character. ASN1_PC_STRING (65): Represents a string of characters in the PC cha…

ASN1_BCD_STRING can be just IA5String with a constraint attached... Your time types can be just an INTEGER with a constraint attached... (In Heimdal we use INTEGER constraints to pick a representation in the programming language.) E.g., -- 64-bit signed count of seconds where 0 is the Unix epoch ASN1_UTC_TIMESTAMP ::= INTEGER (-18446744073709551616..18446744073709551615) ASN1_OUT_OF_BAND can just be a NULL with an AP…

> ASN1_BCD_STRING can be just IA5String with a constraint attached...

The abstract meaning matches, but the format is differently.

> ASN1_OUT_OF_BAND can just be a NULL with an APPLICATION tag or whatever

There are some uses of having a dedicated "out of band" type, such as being able to find them regardless of the schema (e.g. it might be used by a protocol that can use data with any schemas, but allows out of band data with any schema for some reason, and might want to modify the representations of out of band data when sending it to someone else).

> ASN1_IDENTIFIED_DATA... ASN.1 has EMBEDDED-PDV, open types, and the TYPE-IDENTIFIER class -- there are many ways to do this in ASN.1

EMBEDDED-PDV and those other things are different situations than I am doing, although it is similar, the use is not quite the same. ASN1_IDENTIFIED_DATA is simpler in some ways but also allows some things that EMBEDDED-PDV does not do.

Programs can also use ASN1_IDENTIFIED_DATA to identify the schema of a file that uses this type (and potentially be able to e.g. uncompress or decrypt it; this is the reason why the identifiers are allowed to be sequences and not only plain OIDs), or a part of another file.

> ASN1_REFERENCE is... interesting. I've not needed it, but some RPC protocols support intra-payload and even circular references, so if you have a need for that (hopefully you don't), then your ASN1_REFERENCE would be useful indeed.

Yes, it is what I thought too. So far I have not needed it either, but it might sometimes be useful.

> ASN1_RATIONAL is just a tagged sequence of numerator and denominator, with a constraint...

It can be defined as such in standard ASN.1, and the format is the same as that, but the abstract meaning is different. There is also a further constraint for the canonical form.

(Currently, the only place I have used this type is the tempo ratio in the .BGM lumps in Super ZZ Zero, but it would have other uses too, such as when converting data from other formats that have a rational number type.)

> OBJECT IDENTIFIER RELATIVE TO is just a CHOICE of OBJECT IDENTIFIER and RELATIVE IDENTIFIER.

It can be implemented that way in standard ASN.1 and has the same DER representation as your described type, although the abstract meaning is essentially the same as OBJECT IDENTIFIER and there is an additional constraint in the canonical form (as far as I know, this additional constraint cannot be written in standard ASN.1, but Super ZZ Zero cares about it being in canonical form (except for sound card identifiers in .BGM lumps, but this is an implementation detail for that specific part of the program)).

> I had kind of expected a subset of x.680.

Currently I am not using the schema format for ASN.1X (nor do I use the schema format of standard ASN.1); if someone else does then they might implement a variant of X.680 for use with ASN.1X. I probably would remove some stuff (and add some stuff) if I did make a variant, though.

(The use of ASN.1X is also not defined for JER, XER, OER, etc; if someone needs to, then they might do that.)

Re: Why I stopped using JSON for my APIs

#235

Post by MobiusHorizons »

almosthere wrote: post #191
MobiusHorizons wrote: post #101

Earlier quoted context omitted.

Sounds like this introduced behavior changes. How did you evaluate if the new behavior was desirable or not? I’ve definitely run into cases where the missing fields were load bearing in ways the types would not suggest, so I never take it for granted that type error in prod code = bug

it was desirable because our reason for the conversion was subtle bugs all over the place where data was disappearing.

Makes sense. That sounds like a good reason to do it. Unfortulately I've also seen people try to add typescript or various linters without adequate respect for the danger associated with changing code that seems to be working but looks like a bug especially when it requires manual testing to verify.

Re: Why I stopped using JSON for my APIs

#237

Post by morshu9001 »

cryptonector wrote: post #223
morshu9001 wrote: post #221

Earlier quoted context omitted.

Not sure what this is. Transcoding to/from JSON is something protobuf does easily, but this readme is about a lot more than that.

Yes, it's about a lot more than that. It's about automatically and recursively encoding/decoding through "typed hole". A typed hole is where you have a struct with one field that denotes the type of the other, and the other is basically a byte string whose value is an encoding of a value of a type identified by the other field. Typed holes are surprisingly common in protocols. Typically you first decode the outer val…

Why isn't the other value just a oneof? I get if your holed value is passthru data encoded in some special way that isn't standard asn1 or proto, but at that point it's heavily application-dependent and not really the outer protocol's job to support.

Re: Why I stopped using JSON for my APIs

#238

Post by cryptonector »

morshu9001 wrote: post #237
cryptonector wrote: post #223

Earlier quoted context omitted.

Yes, it's about a lot more than that. It's about automatically and recursively encoding/decoding through "typed hole". A typed hole is where you have a struct with one field that denotes the type of the other, and the other is basically a byte string whose value is an encoding of a value of a type identified by the other field. Typed holes are surprisingly common in protocols. Typically you first decode the outer val…

Why isn't the other value just a oneof? I get if your holed value is passthru data encoded in some special way that isn't standard asn1 or proto, but at that point it's heavily application-dependent and not really the outer protocol's job to support.

You can do CHOICE in ASN.1, yes, and you can even make it an extensible CHOICE. In that case the tag is the type determinant, and it looks a lot like a typed hole. But! sometimes you want a typed hole where the type determinant is something like a URN, or a URI, or some other type where the value space is a) large, and b) structured so you can avoid needing a registry. And sometimes the protocol you're writing inherently can't have a type registry -- think of an RPC layer where you have headers that provide things like authentication and negotiation of things, session-like things, while the application provides the procedures (the 'P' in RPC) and so you need to identify the application without a registry of oneof tags.

Re: Why I stopped using JSON for my APIs

#239

Post by cryptonector »

zzo38computer wrote: post #231
morshu9001 wrote: post #216

Earlier quoted context omitted.

Oh gosh. Fair enough that this exists and something uses it, but I'd absolutely want to handle that on the ends only, not get asn1 involved in parsing it.

ASN.1 does not necessarily need to get involved in parsing the values; for some applications doing so is unnecessary anyways (this is true for many fields of many types and not only this one, though). ASN.1 will need to be involved in parsing the framing; whether or not it is involved in parsing the values depends on whether the application requires it for that specific value (for example, it is commonly not necessar…

Correct, ASN.1 does not tell you how to implement all its semantics. You can totally have tooling that exposes a GeneralString's blob payload to the application and lets the app handle the codeset switching aspects of GeneralString.

I want to object that anyone who really has to handle multi-codeset GeneralString values would want a better library but...

...what would that be if not a converter to/from Unicode? But what if one really wants an array/list of {codeset, string} pairs? At that point open-coding support for those escapes is probably just as well since that one might have the only application in the world that wants that!!

:laugh:

Re: Why I stopped using JSON for my APIs

#240

Post by cryptonector »

zzo38computer wrote: post #234
cryptonector wrote: post #215

Earlier quoted context omitted.

ASN1_BCD_STRING can be just IA5String with a constraint attached... Your time types can be just an INTEGER with a constraint attached... (In Heimdal we use INTEGER constraints to pick a representation in the programming language.) E.g., -- 64-bit signed count of seconds where 0 is the Unix epoch ASN1_UTC_TIMESTAMP ::= INTEGER (-18446744073709551616..18446744073709551615) ASN1_OUT_OF_BAND can just be a NULL with an AP…

> ASN1_BCD_STRING can be just IA5String with a constraint attached... The abstract meaning matches, but the format is differently. > ASN1_OUT_OF_BAND can just be a NULL with an APPLICATION tag or whatever There are some uses of having a dedicated "out of band" type, such as being able to find them regardless of the schema (e.g. it might be used by a protocol that can use data with any schemas, but allows out of band…

I couldn't find anything about "Super ZZ Zero". Is this open source? Do you have a link?
Post reply on HN