Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I stopped using JSON for my APIs

aloisdeniel.com

Post reply on HN
Print view View original

171–180 of 245 posts

Re: Why I stopped using JSON for my APIs

#171

Post by gipp »

dfabulich wrote: post #68

> With JSON, you often send ambiguous or non-guaranteed data. You may encounter a missing field, an incorrect type, a typo in a key, or simply an undocumented structure. With Protobuf, that’s impossible. Everything starts with a .proto file that defines the structure of messages precisely. This deeply misunderstands the philosophy of Protobuf. proto3 doesn't even support required fields. https://protobuf.dev/best-pra…

I think the OP meant something far simpler (and perhaps less interesting), which is that you simply cannot encounter key errors due to missing fields, since all fields are always initialized with a default value when deserializing. That's distinct from what a "required" field is in protobuf

Re: Why I stopped using JSON for my APIs

#172

Post by sevensor »

connicpu wrote: post #30
pzmarzly wrote: post #7

> With Protobuf, that’s impossible. Unless your servers and clients push at different time, thus are compiled with different versions of your specs, then many safety bets are off. There are ways to be mostly safe (never reuse IDs, use unknown-field-friendly copying methods, etc.), but distributed systems are distributed systems, and protobuf isn't a silver bullet that can solve all problems on author's list. On the u…

Regardless of whether you use JSON or Protobuf, the only way to be safe from version tears in your serialization format is to enforce backwards compatibility in your CI pipeline by testing the new version of your service creates responses that are usable by older versions of your clients, and vice versa.

Yeah, no discussion of this topic is complete without bringing up schema evolution. There’s a school of thought that holds this is basically impossible and the right thing to do is never ever make a breaking change. Instead, allow new fields to be absent and accept unrecognized fields always. I think this is unsustainable and hard to reason about.

I have not found it difficult to support backwards compatibility with explicit versioning, and the only justification I’ve seen for not doing it is that it’s impossible to coordinate between development teams. Which I think is an indictment of how the company is run more than anything else.

Re: Why I stopped using JSON for my APIs

#174

Post by bdavisx »

duped wrote: post #85
barremian wrote: post #56

Earlier quoted context omitted.

> When judging which alternative will succeed, lower perceived human cost beats lower machine cost every time. Yup this is it. No architect considers using protos unless there is an explicit need for it. And the explicit need is most times using gRPC. Unless the alternative allows for zero cost startup and debugging by just doing `console.log()`, they won't replace JSON any time soon. Edit: Just for context, I'm not…

Print debugging is fine and all but I find that it pays massive dividends to learn how to use a debugger and actually inspect the values in scope rather than guessing which are worth printing. It also is useless when you need to debug a currently running system and can't change the code. And since you need to translate it anyway, there's not much benefit in my mind to using something like msgpack which is more compac…

Debuggers are great when you can use them. Where I work (financial/insurance) we are not allowed to debug on production servers. I would guess that's true in a lot of high security environments.

So the skill of knowing how to "println" debug is still very useful.

Re: Why I stopped using JSON for my APIs

#175

Post by orphea »

gipp wrote: post #171
dfabulich wrote: post #68

> With JSON, you often send ambiguous or non-guaranteed data. You may encounter a missing field, an incorrect type, a typo in a key, or simply an undocumented structure. With Protobuf, that’s impossible. Everything starts with a .proto file that defines the structure of messages precisely. This deeply misunderstands the philosophy of Protobuf. proto3 doesn't even support required fields. https://protobuf.dev/best-pra…

I think the OP meant something far simpler (and perhaps less interesting), which is that you simply cannot encounter key errors due to missing fields, since all fields are always initialized with a default value when deserializing. That's distinct from what a "required" field is in protobuf

Depending on the language/library, you can get exactly the same behavior with JSON.

Re: Why I stopped using JSON for my APIs

#177

Post by mekoka »

duped wrote: post #170
themafia wrote: post #122

Earlier quoted context omitted.

> rather than guessing I'm not guessing. I'm using my knowledge of the program and the error together to decide what to print. I never find the process laborious and I almost always get the right set of variables in the first debug run. The only time I use a debugger is when working on someone else's code.

That's just an educated guess. You can also do it with a debugger.

The debugger is fine, but it's not the key to unlock some secret skill level that you make it out to be. https://lemire.me/blog/2016/06/21/i-do-not-use-a-debugger/

Re: Why I stopped using JSON for my APIs

#178

Post by morshu9001 »

almosthere wrote: post #92
morshu9001 wrote: post #44

Earlier quoted context omitted.

I've gone the all-JSON route many times, and pretty soon it starts getting annoying enough that I lament not using protos. I'm actually against static types in languages, but the API is one place they really matter (the other is the DB). Google made some unforced mistakes on proto usability/popularity though.

why are you against static types in languages? I once converted a fairly large JS codebase to TS and I found about 200 mismatching names/properties all over the place. Tons of properties we had nulls suddenly started getting values.

It costs time, distracts some devs, and adds complexity for negligible safety improvement. Especially if/when the types end up being used everywhere because managers like that metric. I get using types if you have no tests, but you really need tests either way. I've done the opposite migration before, TS to JS.

Oh I forgot to qualify that I'm only talking about high level code, not things that you'd use C or Rust for. But part of the reason those langs have static types is they need to know sizes on stack at compile time.

Re: Why I stopped using JSON for my APIs

#179

Post by morshu9001 »

whatevaa wrote: post #137
cryptonector wrote: post #112

Earlier quoted context omitted.

I've been working on and with Kerberos and PKIX for decades. I don't find ASN.1 to be a problem as long as you have good tooling or are willing to build it. The specs are a pleasure to read -- clear, concise, precise, and approachable (once you have a mental model for it anyways). Of course, I am an ASN.1 compiler maintainer, but hey, I had to become one because the compiler I was using was awesome but not good enoug…

You just said it. You had to become compiler maintainer to make it good enough.

This was the main reason. The asn.1 language has a ton of unnecessary features that make it harder to implement, but the stuff I dealt with was using those features so I couldn't just ignore it. I didn't write a compiler but did hack around some asn1c outputted code to make it faster for our use case. And had to use asn1c in the first place because there was no complete Rust asn1 compiler at the time, though I tried DIY'ing it and gave up.

I also remember it being complicated to use, but it's been too long to recall why exactly, probably the feature bloat. Once I used proto3, I realized it's all you need.

Re: Why I stopped using JSON for my APIs

#180

Post by Pet_Ant »

dfabulich wrote: post #68

> With JSON, you often send ambiguous or non-guaranteed data. You may encounter a missing field, an incorrect type, a typo in a key, or simply an undocumented structure. With Protobuf, that’s impossible. Everything starts with a .proto file that defines the structure of messages precisely. This deeply misunderstands the philosophy of Protobuf. proto3 doesn't even support required fields. https://protobuf.dev/best-pra…

> Protobuf clients need to be written defensively, just like JSON API clients.

Oof. I'd rather just version the endpoints and have required fields. Defensive is error-prone, and verbose, harder to reason about, and still not guaranteed. It really feels like an anti-pattern.

Post reply on HN