Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I stopped using JSON for my APIs

aloisdeniel.com

Post reply on HN
Print view View original

211–220 of 245 posts

Re: Why I stopped using JSON for my APIs

#211

Post by generativenoise »

theshrike79 wrote: post #154
pyrolistical wrote: post #24

Compressed JSON is good enough and requires less human communication initially. Sure it will blow up in your face when a field goes missing or value changes type. People who advocate paying the higher cost ahead of time to perfectly type the entire data structure AND propose a process to do perform version updates to sync client/server are going to lose most of the time. The zero cost of starting with JSON is too com…

Also: debugging You (a human) can just open a JSON request or response and read what's in it. With protobuf you need to build or use tooling that can see what's going on.

It is only "human readable" since our tooling is so bad and the lowest common denominator tooling we have can dump out a sequence of bytes as ascii/utf-8 text somewhat reliably.

One can imagine a world where the lowest common denominator format being a richer structured binary format where every system has tooling to work with it out of the box and that would be considered human readable.

Re: Why I stopped using JSON for my APIs

#212

Post by cryptonector »

morshu9001 wrote: post #179
whatevaa wrote: post #137

Earlier quoted context omitted.

You just said it. You had to become compiler maintainer to make it good enough.

This was the main reason. The asn.1 language has a ton of unnecessary features that make it harder to implement, but the stuff I dealt with was using those features so I couldn't just ignore it. I didn't write a compiler but did hack around some asn1c outputted code to make it faster for our use case. And had to use asn1c in the first place because there was no complete Rust asn1 compiler at the time, though I tried…

> The asn.1 language has a ton of unnecessary features that make it harder to implement

Only if you want to implement them. You could get quite far with just a subset of UNIVERSAL types, including UTF8String, SEQUENCE/SET, SEQUENCE OF / SET OF, etc. There's a ton of features in x.680 you can easily drop.

I've implemented a subset of x.681, x.682, and x.683 to get automatic, recursive decoding through all typed holes in PKIX certificates, CRLs, CSRs, etc. Only a subset, and it got me quite far. I had a pretty good open source x.680 implementation to build on.

This is the story of how Heimdal's authors wrote its ASN.1 compiler: they wanted tooling, there wasn't a good option, they built enough for PKIX and Kerberos. They added things as they went along. OpenSSL does not-quite-DER things? Add support in the Heimdal decoder. They hacked a lot of things for a while which I later fixed, like they didn't support DEFAULT, so they changed DEFAULTed members to OPTIONAL, and they hacked IMPLICIT support, which I finished. And so on. It still doesn't have things like REAL (who needs it in security protocols? no one). Its support for GeneralString is totally half-assed just like... MIT Kerberos, OpenSSL, etc. We do what we need to. Someone could take that code, polish it up, add features, support more programming languages, and make some good money. In fact, Fabrice Belllard has his own not-open-source, commercial ASN.1 compiler and stack, and it must be quite good -- very smart!

Re: Why I stopped using JSON for my APIs

#213

Post by cryptonector »

zzo38computer wrote: post #209
cryptonector wrote: post #206

Earlier quoted context omitted.

> However, Unicode is not a very good character set, and they should not force or expect you to use it. Unicode is an excellent character set, and for 99% of cases (much more probably) it's absolutely the best choice. So one should choose Unicode (and UTF-8) in all cases unless there is an excellent reason to do otherwise. As time passes there will be fewer and fewer cases where Unicode is not sufficient, so really w…

> Have you written up anything about ASN.1X anywhere? I'd love to take a look. ASN1_BCD_STRING (64): Represents a string with the following characters: "0123456789*#+-. " (excluding the quotation marks). Each octet encodes two characters, where the high nybble corresponds to the first character and the low nybble corresponds to the second character. ASN1_PC_STRING (65): Represents a string of characters in the PC cha…

Re: ASN1_KEY_VALUE_LIST, why not just do something like:

  ValueList ::= SET OF KeyValue

  KeyValue ::= SEQUENCE { key MyKeyType, value MyValueType }

  MyKeyType   ::= ... -- whatever you want here
  MyValueType ::= ... -- ditto

?

Re: Why I stopped using JSON for my APIs

#214

Post by cryptonector »

zzo38computer wrote: post #209
cryptonector wrote: post #206

Earlier quoted context omitted.

> However, Unicode is not a very good character set, and they should not force or expect you to use it. Unicode is an excellent character set, and for 99% of cases (much more probably) it's absolutely the best choice. So one should choose Unicode (and UTF-8) in all cases unless there is an excellent reason to do otherwise. As time passes there will be fewer and fewer cases where Unicode is not sufficient, so really w…

> Have you written up anything about ASN.1X anywhere? I'd love to take a look. ASN1_BCD_STRING (64): Represents a string with the following characters: "0123456789*#+-. " (excluding the quotation marks). Each octet encodes two characters, where the high nybble corresponds to the first character and the low nybble corresponds to the second character. ASN1_PC_STRING (65): Represents a string of characters in the PC cha…

What are the numbers in parenthesis? UNIVERSAL tag values?

Re: Why I stopped using JSON for my APIs

#215

Post by cryptonector »

zzo38computer wrote: post #209
cryptonector wrote: post #206

Earlier quoted context omitted.

> However, Unicode is not a very good character set, and they should not force or expect you to use it. Unicode is an excellent character set, and for 99% of cases (much more probably) it's absolutely the best choice. So one should choose Unicode (and UTF-8) in all cases unless there is an excellent reason to do otherwise. As time passes there will be fewer and fewer cases where Unicode is not sufficient, so really w…

> Have you written up anything about ASN.1X anywhere? I'd love to take a look. ASN1_BCD_STRING (64): Represents a string with the following characters: "0123456789*#+-. " (excluding the quotation marks). Each octet encodes two characters, where the high nybble corresponds to the first character and the low nybble corresponds to the second character. ASN1_PC_STRING (65): Represents a string of characters in the PC cha…

ASN1_BCD_STRING can be just IA5String with a constraint attached...

Your time types can be just an INTEGER with a constraint attached... (In Heimdal we use INTEGER constraints to pick a representation in the programming language.) E.g.,

  -- 64-bit signed count of seconds where 0 is the Unix epoch
  ASN1_UTC_TIMESTAMP ::= INTEGER (-18446744073709551616..18446744073709551615)
ASN1_OUT_OF_BAND can just be a NULL with an APPLICATION tag or whatever:

  Out-of-Band ::= [APPLICATION 100] NULL
or maybe an ENUMERATED or BIT STRING with named bits to indicate what kind of thing is referenced out of band. You might even use this with a SEQUENCE type instead where one member identifies an out of band datum as an index, and the other identifies the kind.

ASN1_REFERENCE is... interesting. I've not needed it, but some RPC protocols support intra-payload and even circular references, so if you have a need for that (hopefully you don't), then your ASN1_REFERENCE would be useful indeed.

ASN1_IDENTIFIED_DATA... ASN.1 has EMBEDDED-PDV, open types, and the TYPE-IDENTIFIER class -- there are many ways to do this in ASN.1. See https://github.com/heimdal/heimdal/blob/master/lib/asn1/READ...

ASN1_RATIONAL is just a tagged sequence of numerator and denominator, with a constraint that the denominator must not be zero.

OBJECT IDENTIFIER RELATIVE TO is just a CHOICE of OBJECT IDENTIFIER and RELATIVE IDENTIFIER.

Re: SDER... yeah, so Heimdal's codec produces DER but accepts a subset of BER for interop with OpenSSL and others. If you really want streaming then you'll want a variant of OER with fixed-length lengths (which IMO OER should have had, dammit), which then looks a lot like XDR but with different alignment and more types.

I had kind of expected a subset of x.680.

Re: Why I stopped using JSON for my APIs

#216

Post by morshu9001 »

cryptonector wrote: post #199
morshu9001 wrote: post #196

Earlier quoted context omitted.

I'm still confused how these ISO 2022 strings even work, and the ASN1 docs discourage using the UniversalString and GraphicString types. All these different string types are intimidating if I just want unicode/ascii, and even if I were using an obscure encoding, I'd use generic bytes instead of wanting asn1 to care about it.

GeneralString relies on control characters to "load" character sets into the C0 and C1 registers. This is madness -- specifically it's pre-Unicode madness, but before Unicode it made sense.

Oh gosh. Fair enough that this exists and something uses it, but I'd absolutely want to handle that on the ends only, not get asn1 involved in parsing it.

Re: Why I stopped using JSON for my APIs

#217

Post by morshu9001 »

cryptonector wrote: post #204
morshu9001 wrote: post #203

Earlier quoted context omitted.

Protobuf ended up having more tooling in the end though, and it didn't take very long to get there. This is like how JSON replaced XML for many use cases.

If they had put the same energy towards building tooling for an existing IDL/codec then they would have had strictly less work to do. Besides being inefficient in the use of their resources they also saddled us with 15th system (probably more like a 25th system, but you get the reference), and a poor one at that. There is really nothing much good to say about PB.

I rely on protos for lots of stuff at work and honestly couldn't imagine having to do all this in ASN.1, even if tooling were completely solved.

Re: Why I stopped using JSON for my APIs

#218

Post by cryptonector »

morshu9001 wrote: post #217
cryptonector wrote: post #204

Earlier quoted context omitted.

If they had put the same energy towards building tooling for an existing IDL/codec then they would have had strictly less work to do. Besides being inefficient in the use of their resources they also saddled us with 15th system (probably more like a 25th system, but you get the reference), and a poor one at that. There is really nothing much good to say about PB.

I rely on protos for lots of stuff at work and honestly couldn't imagine having to do all this in ASN.1, even if tooling were completely solved.

I use both (and JSON, and I've used XML, and I've used XDR, and ...). Check this out and weep for not having anything like this for PB: https://github.com/heimdal/heimdal/blob/master/lib/asn1/READ...

Re: Why I stopped using JSON for my APIs

#219

Post by cryptonector »

morshu9001 wrote: post #216
cryptonector wrote: post #199

Earlier quoted context omitted.

GeneralString relies on control characters to "load" character sets into the C0 and C1 registers. This is madness -- specifically it's pre-Unicode madness, but before Unicode it made sense.

Oh gosh. Fair enough that this exists and something uses it, but I'd absolutely want to handle that on the ends only, not get asn1 involved in parsing it.

Oh GeneralString is madness. It's pre-Unicode madness. It exists because Unicode didn't exist in 1984, but people still wanted to be able to exchange text in multiple scripts, which necessitated being able to "switch codesets" in the middle. It's... yeah, it's.. it's nuts. I've _not_ implemented GeneralString, and practically no one needs to even when specs say to. E.g., in Kerberos the strings are GeneralString, but all the implementations just-send-8 and do not attempt to interpret any codeset switching escapes.

Re: Why I stopped using JSON for my APIs

#220

Post by wallaconno »

My criticism is that protobuf is a premature optimization for most projects. I want to love protobuf, but it usually slows down my development pace and it’s just not worth it for most web / small data projects.

Distributing the contract with lock-step deploys between services is a bit much. JSON parsing time and size are not usually key factors in my projects. Protobuf doesn’t pose strict requirements for data validation, so I have to do that anyway. Losing data readability in transit is a huge problem.

Protobuf seems like it would be amazing in situations where data is illegible and tightly coupled such as embedded CAN or radio.

Post reply on HN