Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Object – A JSON alternative data serialization format

internetobject.org

Post reply on HN
Print view View original

71–80 of 83 posts

Re: Internet Object – A JSON alternative data serialization format

#71

Post by samhw »

throwaway81523 wrote: post #65
snidane wrote: post #61

Earlier quoted context omitted.

It's just a matter of parser implementation. These changes are backwards compatible. If python decided to add support for comments and trailing commas in json.loads, that would become the new standard, at least for data scientists, not for web devs. All the other ones would then follow.

Now whatever generates your data has to know what parser is going to read the data. The parser can't tell right away whether the data has those trailing commas. They are optional, so they might not start appearing until after gigabytes of output have gone by. So you can't count on a quick error message in the event of a version mismatch.

If you have gigabytes of handwritten JSON (if it's not handwritten, trailing vs non-trailing commas surely don't matter), then I feel like you're doing something wrong.

Though I'm sure someone's going to step in and say "Have you not heard of [stupendously niche use case]? Are you living under a rock!?" etc etc ;)

Re: Internet Object – A JSON alternative data serialization format

#72

Post by fstrthnscnd »

samhw wrote: post #9
flqn wrote: post #7

I'm sceptical about the value proposition of this without seeing much more than a simple example that offers little over existing hypermedia+json/csv practices. If a compact columnar representation is what you're after to avoid having to repeat every field name in an array of objects (which CSV is good for) but you don't want to give up the ability to include metadata in your JSON, there are a ton of different ways f…

> If a compact columnar representation is what you're after to avoid having to repeat every field name in an array of objects (which CSV is good for) Plus, as I wrote elsewhere, gzipping your JSON will result in essentially "avoiding having to repeat every field name" by dictionary coding it. The only case in which that wouldn't be true is when dealing with extremely unusual and heteromorphic data, but then this form…

> Plus, as I wrote elsewhere, gzipping your JSON will result in essentially "avoiding having to repeat every field name" by dictionary coding it.

Gzipping indeed helps in getting mostly back the space taken by the field names, but a parser will still have to parse these strings. On a large document, this might have a performance impact.

One good side of having the field names however is that one can reorder them adlib.

Re: Internet Object – A JSON alternative data serialization format

#73

Post by throwaway81523 »

samhw wrote: post #71
throwaway81523 wrote: post #65

Earlier quoted context omitted.

Now whatever generates your data has to know what parser is going to read the data. The parser can't tell right away whether the data has those trailing commas. They are optional, so they might not start appearing until after gigabytes of output have gone by. So you can't count on a quick error message in the event of a version mismatch.

If you have gigabytes of handwritten JSON (if it's not handwritten, trailing vs non-trailing commas surely don't matter), then I feel like you're doing something wrong. Though I'm sure someone's going to step in and say "Have you not heard of [stupendously niche use case]? Are you living under a rock!?" etc etc ;)

It's silly to not write your software to handle every possible input instead of every input you think is likely based on some predictions about humans. Failure to do that is why YAML is so broken.

JSON isn't a format conducive to handwriting even if it probably should have made more accomodations for that at the start. Right now it can't even handle trailing newlines. But if you want to fix that, call it something different (maybe even JSON2), for heaven's sake.

I doubt anyone would handwrite an entire gigabyte JSON document, but they might hand-edit a machine-generated one to make a change someplace in it, end up putting in a trailing comma, and have the document pass their local tests but crash a remote parser.

Re: Internet Object – A JSON alternative data serialization format

#75

Post by musicale »

Looks like CSV with a schema, which is OK but can become unreadable if your field (column) space is large and sparse (imagine 50 different optional fields of the same type.)

I still kind of like classic NeXT (and pre-XML OS X) property lists.

GNUstep seems address some of their limitations:

http://wiki.gnustep.org/index.php/Property_Lists https://everything.explained.today/Property_list/

I think Apple probably erred in switching to XML.

Re: Internet Object – A JSON alternative data serialization format

#76

Post by tzekid »

Waterluvian wrote: post #30

We are paying a cost in clarity, human editability, and further splintering of formats. Everything is a trade off. So what do we get in trade for those rather large costs? 40% bandwidth savings might be worth it. But what are the gzipped comparisons?

Compression and decompression (gzip) takes computing power and RAM. The resulting JSON (in memory) is still harder to parse because of the required field names...

Re: Internet Object – A JSON alternative data serialization format

#78

Post by samhw »

throwaway81523 wrote: post #73
samhw wrote: post #71

Earlier quoted context omitted.

If you have gigabytes of handwritten JSON (if it's not handwritten, trailing vs non-trailing commas surely don't matter), then I feel like you're doing something wrong. Though I'm sure someone's going to step in and say "Have you not heard of [stupendously niche use case]? Are you living under a rock!?" etc etc ;)

It's silly to not write your software to handle every possible input instead of every input you think is likely based on some predictions about humans. Failure to do that is why YAML is so broken. JSON isn't a format conducive to handwriting even if it probably should have made more accomodations for that at the start. Right now it can't even handle trailing newlines. But if you want to fix that, call it something di…

> Though I'm sure someone's going to step in and say "Have you not heard of [stupendously niche use case]?"

> they might hand-edit a machine-generated one to make a change someplace in it

Ah, there's [stupendously niche use case] ;)

Seriously, though, I do agree with your point that good software should handle every edge case. I'm not arguing that.

But the case for having trailing commas does seem to be generally predicated on handwritten JSON, so I'm saying it's _unlikely_ it would be used in that way, and therefore that such failures would be rare and thus not a very grave counterargument.

Re: Internet Object – A JSON alternative data serialization format

#79

Post by samhw »

fstrthnscnd wrote: post #72
samhw wrote: post #9

Earlier quoted context omitted.

> If a compact columnar representation is what you're after to avoid having to repeat every field name in an array of objects (which CSV is good for) Plus, as I wrote elsewhere, gzipping your JSON will result in essentially "avoiding having to repeat every field name" by dictionary coding it. The only case in which that wouldn't be true is when dealing with extremely unusual and heteromorphic data, but then this form…

> Plus, as I wrote elsewhere, gzipping your JSON will result in essentially "avoiding having to repeat every field name" by dictionary coding it. Gzipping indeed helps in getting mostly back the space taken by the field names, but a parser will still have to parse these strings. On a large document, this might have a performance impact. One good side of having the field names however is that one can reorder them adli…

That's true, but the main argument made by the website is about the space advantage, so it's very relevant that that space advantage is basically nullified by the widespread use of compression.

If your worry is parsing speed, then JSON not only has battle-tested parsers, but also has SIMD-assisted parsers which can process gigabytes a second on a single core (e.g. https://github.com/simdjson/simdjson). It would take Internet Object years to develop parsers as performant as that, even if it did, by some miracle, achieve wide uptake. So the notional advantage afforded by not having keys on each row is neither here nor there.

And incidentally, as someone who's written a handful of parsers, I suspect that this scheme would not be particularly easy to parse. You need lookahead because of optional fields, as well as maintaining state and a lookup table for mapping positions to keys, etc. I can draw up a quick parser in pseudocode or Python to explain, if you disagree.

Post reply on HN