Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I stopped using JSON for my APIs

aloisdeniel.com

Post reply on HN
Print view View original

151–160 of 245 posts

Re: Why I stopped using JSON for my APIs

#151

Post by Ghoelian »

bloppe wrote: post #140
dfabulich wrote: post #68

> With JSON, you often send ambiguous or non-guaranteed data. You may encounter a missing field, an incorrect type, a typo in a key, or simply an undocumented structure. With Protobuf, that’s impossible. Everything starts with a .proto file that defines the structure of messages precisely. This deeply misunderstands the philosophy of Protobuf. proto3 doesn't even support required fields. https://protobuf.dev/best-pra…

Skew is an inherent problem of networked systems no matter what the encoding is. But, once the decoding is done, assuming there were no decoding errors in either case, at least with protobuf you have a statically typed object. You could also just validate the JSON payload, but most people don't bother. And then they just pass the JSON blob around to all sorts of functions, adding, modifying, and removing fields until…

The convention at every company I've worked at was to use DTO's. So yes, JSON payloads are in fact validated, usually with proper type validation as well (though unfortunately that part is technically optional since we work in php).

Usually it's not super strict, as in it won't fail if a new field suddenly appears (but will if one that's specified disappears), but that's a configuration thing we explicitly decided to set this way.

Re: Why I stopped using JSON for my APIs

#152

Post by JodieBenitez »

davedx wrote: post #124

"Ultra-efficient" Searched the article, no mention of gzip, and how most of the time all that text data (html, js and css too!) you're sending over the wire will be automatically compressed to...... an efficient binary format! So really, the author should compare protobufs to gzipped JSON

This is so obvious to me... JSON vs. JSON + mod_deflate is just night and day.

Re: Why I stopped using JSON for my APIs

#153

Post by theshrike79 »

chillfox wrote: post #128
davedx wrote: post #124

"Ultra-efficient" Searched the article, no mention of gzip, and how most of the time all that text data (html, js and css too!) you're sending over the wire will be automatically compressed to...... an efficient binary format! So really, the author should compare protobufs to gzipped JSON

Last time I was evaluating different binary serialization formats for an API I was really hoping to get to use one of the cool ones, but gzipped JSON just beat everything and it wasn't even close.

There are some compression formats that perform better than gzip, but it's very dependent on the data you're compressing and your timing requirements (is bandwidth or CPU more important to conserve).

But in the end compressed JSON is pretty good. Not perfect, but good enough for many many things.

Re: Why I stopped using JSON for my APIs

#154

Post by theshrike79 »

pyrolistical wrote: post #24

Compressed JSON is good enough and requires less human communication initially. Sure it will blow up in your face when a field goes missing or value changes type. People who advocate paying the higher cost ahead of time to perfectly type the entire data structure AND propose a process to do perform version updates to sync client/server are going to lose most of the time. The zero cost of starting with JSON is too com…

Also: debugging

You (a human) can just open a JSON request or response and read what's in it.

With protobuf you need to build or use tooling that can see what's going on.

Re: Why I stopped using JSON for my APIs

#155

Post by liampulles »

I know that OpenAPI code gen support is spotty, and that protobuf codegen (in my experience) is quite good, but all of this starts from the idea that the SaaS I'm consuming has actually documented their API properly.

A sizable portion of the integrations I've built have had to be built by hand, because there are inevitable stupid quirks and/or failures I've had to work around. For these usecases, using JSON is preferable, because it is easy for me to see what I have actually been sent, not what the partially up to date spec says I should've been sent.

This is consistent with the idea that communication over the internet should consist of (encrypted and compressed) plain text. It's because human beings are going to have to deal with human reality at the end of the day.

Re: Why I stopped using JSON for my APIs

#156

Post by akoboldfrying »

andersmurphy wrote: post #143
davedx wrote: post #124

"Ultra-efficient" Searched the article, no mention of gzip, and how most of the time all that text data (html, js and css too!) you're sending over the wire will be automatically compressed to...... an efficient binary format! So really, the author should compare protobufs to gzipped JSON

Or streaming brotli/zstd json/html where the compression window can be used for the duration of the connection.

But in that case the server/CDN won't be able to cache the gzipped forms of the individual files -- so probably a win for highly dynamic/user-specific content, but a loss for static or infrequently generated content.

Re: Why I stopped using JSON for my APIs

#157

Post by kragen »

themafia wrote: post #125
theamk wrote: post #81

Earlier quoted context omitted.

ASN.1 has too much stuff. The moment you write "I made ASN.1 decoder/encoder", someone will throw TeletexString or BMPString at it. Or inheritance, as morshu9001 sad. So at this point: - You can support all those features, and your ASM.1 library will be horribly bloated and over-engineered. - You can support your favorite subset, but then you cannot say it's ASN.1 anymore. It will be "ASN.brabel", which only has one…

> someone will throw TeletexString or BMPString ASCII with escapes and UCS-2. > horribly bloated and over-engineered. It's no more or less complicated than XML, JSON or CSV. Which is why you can use ASN.1 to serialize to and from all these formats. ASN.1 provides you an additional level of schema above these. It simply allows you to describe your problem. I find ASN.1 far more sane and useful than something like JSON…

ASN.1 is far, far more complicated than JSON or any particular flavor of CSV, in part because it does provide an extra level of schema that those other formats don't.

Re: Why I stopped using JSON for my APIs

#158

Post by kragen »

ses1984 wrote: post #82
strongpigeon wrote: post #36

Earlier quoted context omitted.

> Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves. Yeah. I do remember a lot of workloads at Google where most of the CPU time was spent serializing/deserializing protos.

I feel like most high throughput distributed systems eventually reach a point where some part of it is constrained by de/serialization. Not much is faster than protobuf except for zero copy formats.

But zero-copy formats like FlatBuffers or Cap'n Proto can be much faster. Like, faster by an arbitrarily large factor, for data at rest.

Re: Why I stopped using JSON for my APIs

#159

Post by kragen »

bloppe wrote: post #18
brabel wrote: post #15

Mandatory comment about ASN.1, a protocol from 1984, already did what Protobuf does, with more flexibility. Yes, it's a bit ugly but if you stick to the DER encoding it's really not worse than Protbuf at all. Check out the Wikipedia example: https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves.

What makes it too complex in your opinion?

Experience with ASN.1.

Re: Why I stopped using JSON for my APIs

#160

Post by kragen »

TillE wrote: post #99
ElectricalUnion wrote: post #91

Earlier quoted context omitted.

Serialization issue. From the Introduction to Cap’n Proto: "Cap’n Proto is INFINITY TIMES faster than Protocol Buffers. (...) there is no encoding/decoding step. The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out". I take it as a rationalization of what OLE Compound File Binary - intern…

Google has a library/format for that too, with FlatBuffers. Different use cases and advantages really, not clearly better/worse.

Kenton Varda also worked on Protobufs at Google before he wrote CapnProto, I think.
Post reply on HN