Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I stopped using JSON for my APIs

aloisdeniel.com

Post reply on HN
Print view View original

221–230 of 245 posts

Re: Why I stopped using JSON for my APIs

#221

Post by morshu9001 »

cryptonector wrote: post #218
morshu9001 wrote: post #217

Earlier quoted context omitted.

I rely on protos for lots of stuff at work and honestly couldn't imagine having to do all this in ASN.1, even if tooling were completely solved.

I use both (and JSON, and I've used XML, and I've used XDR, and ...). Check this out and weep for not having anything like this for PB: https://github.com/heimdal/heimdal/blob/master/lib/asn1/READ...

Not sure what this is. Transcoding to/from JSON is something protobuf does easily, but this readme is about a lot more than that.

Re: Why I stopped using JSON for my APIs

#223

Post by cryptonector »

morshu9001 wrote: post #221
cryptonector wrote: post #218

Earlier quoted context omitted.

I use both (and JSON, and I've used XML, and I've used XDR, and ...). Check this out and weep for not having anything like this for PB: https://github.com/heimdal/heimdal/blob/master/lib/asn1/READ...

Not sure what this is. Transcoding to/from JSON is something protobuf does easily, but this readme is about a lot more than that.

Yes, it's about a lot more than that. It's about automatically and recursively encoding/decoding through "typed hole". A typed hole is where you have a struct with one field that denotes the type of the other, and the other is basically a byte string whose value is an encoding of a value of a type identified by the other field. Typed holes are surprisingly common in protocols. Typically you first decode the outer value, then you inspect the typed hole's type ID field, then you decode the typed hole's value accordingly, and this is code you have to write by hand. Whereas with automatic handling of typed holes just one invocation of the codec is sufficient (as opposed to one codec invocation for the outermost value plus one invocation for every typed hole).

Re: Why I stopped using JSON for my APIs

#224

Post by rerdavies »

chrismorgan wrote: post #189
cosmotic wrote: post #183

Earlier quoted context omitted.

Icons are no longer fixed sizes. They're are numerous dpi/scaling settings even if the "size" doesn't change.

The article goes into that, it’s making a sprite map of at least the expected scaling factors.

There are no "expected" scaling factors anymore.

Re: Why I stopped using JSON for my APIs

#225

Post by squirrellous »

recursivecaveat wrote: post #54

My dream binary format is schema driven, as compact and efficient as Capt Proto or such, but just optionally embeds the entire schema into the message. Then we can write a vim plugin that just opens the file in human readable form without having to fish for the schema. Whenever I am using binary formats, it's because I have a list of millions of objects of the same types. Seems to me that you may as well tack 1KB of…

It’s possible to build this around protobuf. Google has a rich internal protobuf ecosystem that does this and supports querying large amounts of protobuf data without specifying schemas. They are only selectively open sourced. Have a look at riegeli if you are interested.

https://github.com/google/riegeli

Re: Why I stopped using JSON for my APIs

#226

Post by HelloNurse »

eviks wrote: post #168
chrismorgan wrote: post #166

Earlier quoted context omitted.

https://aloisdeniel.com/blog/i-changed-my-mind-about-vector-... seems fairly clearly to be talking about icons of known sizes, in which case that advantage disappears. (I still feel the article is misguided and that the benefit of runtime-determined scaling should have been mentioned, and see no benchmarks supporting its performance theses, and I’d be surprised if the difference was anything but negligible; vector gr…

Are there display pipelines that cache the generated-for-my-device-resolution svgs instead of doing all the slower parsing etc from scratch every time, achieving benefits of both worlds? And you can still have runtime-defined scaling by "just" rebuilding the cache?

The best place for such a cache is a GPU texture, and in a shader that does simple texture mapping instead of rasterizing shapes it would cost more memory reads in exchange for less calculations.

Re: Why I stopped using JSON for my APIs

#227

Post by eliasdejong »

IshKebab wrote: post #200
barremian wrote: post #53

Earlier quoted context omitted.

I think the it-just-works nature and human readability for debugging JSON cannot be overstated. Most people and projects are content to just use JSON even if protos offer some advantages, if not only to save time and resources. Whether the team saves times in the longer when using protos is a question in its own.

There are plenty of it-doesn't-just-work things about JSON though. Sending binary data or 64-bit integers is a huge pain. Or maps with non-string keys, or ordered maps. Plus JSON doesn't scale well with message size because it doesn't use TLV so parsing any of a message requires parsing all of it. It's not some perfect format. That said, I'm disappointed with Protobuf too. Especially the handling of optional/default…

Check out Lite³, a schemaless binary format: https://github.com/fastserial/lite3

Supports 64 bit ints, raw byte datatype, zero-copy parsing, does not require schema and can be converted to JSON for readability while retaining all field names.

Re: Why I stopped using JSON for my APIs

#228

Post by coolThingsFirst »

davedx wrote: post #124

"Ultra-efficient" Searched the article, no mention of gzip, and how most of the time all that text data (html, js and css too!) you're sending over the wire will be automatically compressed to...... an efficient binary format! So really, the author should compare protobufs to gzipped JSON

How on earth would gzipping larger amount of data be more efficient than gzipping smaller amount of data?

Re: Why I stopped using JSON for my APIs

#229

Post by port11 »

thayne wrote: post #150
port11 wrote: post #8

Earlier quoted context omitted.

I think the strict schema of Protobuf might be one of the major improvements, as most APIs don't publish a JSON schema? I've always had to use ajv or superstruct to make sure payloads match a schema, Protobuf doesn't need that (supposedly).

One limitation of protobuf 3 schemas, is they doen't allow required fields. That makes it easier to remove the field in a later version in a backwards compatible way, but sometimes fields really are required, and the message doesn't make any sense without them. Ideally, IMO, if the message is missing those fields, it would fail to parse successfully. But with protobuf, you instead get a default value, which could pot…

Okay, this is a definite issue, you're still stuck validating inputs/outputs.

Re: Why I stopped using JSON for my APIs

#230

Post by ablob »

coolThingsFirst wrote: post #228
davedx wrote: post #124

"Ultra-efficient" Searched the article, no mention of gzip, and how most of the time all that text data (html, js and css too!) you're sending over the wire will be automatically compressed to...... an efficient binary format! So really, the author should compare protobufs to gzipped JSON

How on earth would gzipping larger amount of data be more efficient than gzipping smaller amount of data?

It's a question of entropy. Data is rarely truly random and for larger data there is a lot higher chance of having this "unrandomness" occur.

If your data consists of 4 kilobytes of just 00_01, then you gain a lot by just remembering:

  "write 00_01 2000 times".
Conversely, if the small amount of data is 00_01_00_01_00_01 then using the previous format would yield:

  "write 00_01 3 times"
As you can see, it does not nearly save as much space in comparison with the original data, hence it's less efficient to use the format. The specifics are highly dependent on the compression algorithmm used so take the example with a grain of salt, but I hope it gets the basic idea of why it can be more efficient across.
Post reply on HN