Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I stopped using JSON for my APIs

aloisdeniel.com

Post reply on HN
Print view View original

41–50 of 245 posts

Re: Why I stopped using JSON for my APIs

#41

Post by morshu9001 »

Protos are great. Last time I did a small project in NodeJS, I set up a server that defines the entire API in a .proto and serves each endpoint as either proto or json, depending on the content header. Even if the clients want to use json, at least I can define the whole API in proto spec instead of something like Swagger.

So my question is, why didn't Google just provide that as a library? The setup wasn't hard but wasn't trivial either, and had several "wrong" ways to set up the proto side. They also bait most people with gRPC, which is its own separate annoying thing that requires HTTP/2, which even Google's own cloud products don't support well (e.g App Engine).

P.S. Text proto is also the best static config language. More readable than JSON, less error-prone than YAML, more structure than both.

Re: Why I stopped using JSON for my APIs

#42

Post by zzo38computer »

morshu9001 wrote: post #35
brabel wrote: post #15

Mandatory comment about ASN.1, a protocol from 1984, already did what Protobuf does, with more flexibility. Yes, it's a bit ugly but if you stick to the DER encoding it's really not worse than Protbuf at all. Check out the Wikipedia example: https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves.

ASN.1 is way overengineered to the point of making it hard to support. You don't need inheritance for example.

it is not necessary to use or to implement all of the data types and other features of ASN.1; you can implement only the features that you are using. Since DER uses the same framing for all data types, it is possible to skip past any fields that you do not care about (although in some cases you will still need to check its type, to determine whether or not an optional field is present; fortunately the type can be checked easily, even if it is not a type you implement).

Re: Why I stopped using JSON for my APIs

#43

Post by jtrn »

I like Python-like indentation, but I usually read Python in an IDE or code blocks. JSON in a non-monospace environment might be problematic with some fonts. Hell, I pass JSON around in emails and word processors all the time.

Re: Why I stopped using JSON for my APIs

#44

Post by morshu9001 »

pyrolistical wrote: post #24

Compressed JSON is good enough and requires less human communication initially. Sure it will blow up in your face when a field goes missing or value changes type. People who advocate paying the higher cost ahead of time to perfectly type the entire data structure AND propose a process to do perform version updates to sync client/server are going to lose most of the time. The zero cost of starting with JSON is too com…

I've gone the all-JSON route many times, and pretty soon it starts getting annoying enough that I lament not using protos. I'm actually against static types in languages, but the API is one place they really matter (the other is the DB). Google made some unforced mistakes on proto usability/popularity though.

Re: Why I stopped using JSON for my APIs

#45

Post by wg0 »

Protos don't work out of the box in any browsers as far as I checked last time unless you're willing to deploy a proxy in front to do the translation and it requires extra dependency on the browser as well.

Plus - tooling.

JSON might not be simpler or strict but it gets the job done.

If JSON's size or performance is causing you to go out of business, you surely have bigger problems than JSON.

Re: Why I stopped using JSON for my APIs

#46

Post by morshu9001 »

zzo38computer wrote: post #42
morshu9001 wrote: post #35

Earlier quoted context omitted.

ASN.1 is way overengineered to the point of making it hard to support. You don't need inheritance for example.

it is not necessary to use or to implement all of the data types and other features of ASN.1; you can implement only the features that you are using. Since DER uses the same framing for all data types, it is possible to skip past any fields that you do not care about (although in some cases you will still need to check its type, to determine whether or not an optional field is present; fortunately the type can be che…

Yes but I don't want to worry about what parts of the spec are implemented on each end. If you removed all the unnecessary stuff and formed a new standard, it'd basically be protobuf.

Re: Why I stopped using JSON for my APIs

#47

Post by dhussoe »

brabel wrote: post #17
pzmarzly wrote: post #7

> With Protobuf, that’s impossible. Unless your servers and clients push at different time, thus are compiled with different versions of your specs, then many safety bets are off. There are ways to be mostly safe (never reuse IDs, use unknown-field-friendly copying methods, etc.), but distributed systems are distributed systems, and protobuf isn't a silver bullet that can solve all problems on author's list. On the u…

No type system survives going through a network.

yes, but any sane JSON parsing library (Rust Serde, kotlinx-serialization, Swift, etc.) will raise an error when you have the wrong type or are missing a required field. and any JSON parsing callsite is very likely also an IO callsite so you need to handle errors there anyways, all IO can fail. then you log it or recover or whatever you do when IO fails in some other way in that situation.

this seems like a problem only if you use JSON.parse or json.loads etc. and then just cross your fingers and hope that the types are correct, basically doing the silent equivalent of casting an "any" type to some structure that you assume is correct, rather than strictly parsing (parse, don't validate) into a typed structure before handing that off to other code.

Re: Why I stopped using JSON for my APIs

#48

Post by noctune »

morshu9001 wrote: post #41

Protos are great. Last time I did a small project in NodeJS, I set up a server that defines the entire API in a .proto and serves each endpoint as either proto or json, depending on the content header. Even if the clients want to use json, at least I can define the whole API in proto spec instead of something like Swagger. So my question is, why didn't Google just provide that as a library? The setup wasn't hard but…

You might be interested in https://connectrpc.com/. It's basically what you describe, though it's not clear to me how well supported it is.

Re: Why I stopped using JSON for my APIs

#49

Post by morshu9001 »

noctune wrote: post #48
morshu9001 wrote: post #41

Protos are great. Last time I did a small project in NodeJS, I set up a server that defines the entire API in a .proto and serves each endpoint as either proto or json, depending on the content header. Even if the clients want to use json, at least I can define the whole API in proto spec instead of something like Swagger. So my question is, why didn't Google just provide that as a library? The setup wasn't hard but…

You might be interested in https://connectrpc.com/ . It's basically what you describe, though it's not clear to me how well supported it is.

Yeah that one looked good. I don't remember why I didn't use it that time, maybe just felt it was easy enough to DIY that I didn't feel like using another dep (given that I already knew express and proto in isolation). The thing is, Google themselves had to lead the way on this if they wanted protobuf to be mainstream like JSON.

Re: Why I stopped using JSON for my APIs

#50

Post by dhussoe »

I like that JSON parsing libraries (Serde etc.) allow you to validate nullability constraints at parse-time. Protobuf's deliberate lack of support for required fields means that either you kick that down to every callsite, or you need to build another parsing layer on top of the generated protobuf code.

Now, there is a serde_protobuf (I haven't used it) that I assume allows you to enforce nullability constraints but one of the article's points is that you can use the generated code directly and:

> No manual validation. No JSON parsing. No risk of type errors.

But this is not true—nullability errors are type errors. Manual validation is still required (except you should parse, not validate) to make sure that all of the fields your app expects are there in the response. And "manual" validation (again, parse don't validate) is not necessary with any good JSON parsing library, the library handles it.

Post reply on HN