Live data from Hacker News

Internet Object – A JSON alternative data serialization format

internetobject.org

11–20 of 83 posts

Re: Internet Object – A JSON alternative data serialization format

#12
As far I can see "IO" addresses the size issue, which is indeed a compression issue for the most part.

For a broader take on an alternative, there is concise encoding Concise Encoding [1][2], which I believe addresses a few more issues with existing encodings (clear spec, schema not an afterthought, native support for a variety of data structures, security, ...).

[1] https://concise-encoding.org/ [2] The author gave a presentation on it here: https://www.youtube.com/watch?v=_dIHq4GJE14

Re: Internet Object – A JSON alternative data serialization format

#13
Since strings don't need to be quoted, what happens during deserialization if you want the string "T"? Does this lead to the equivalent of the Norway-Problem of YAML [0]?

Is the space between the key and the type necessary? If not, how to distinguish between objects and types?

Does the validation offer some form of unions or mutual exclusion?

[0]: https://hitchdev.com/strictyaml/why/implicit-typing-removed/

Re: Internet Object – A JSON alternative data serialization format

#14
post #7

I'm sceptical about the value proposition of this without seeing much more than a simple example that offers little over existing hypermedia+json/csv practices. If a compact columnar representation is what you're after to avoid having to repeat every field name in an array of objects (which CSV is good for) but you don't want to give up the ability to include metadata in your JSON, there are a ton of different ways f…

I agree. CSV + Metadata/field types (which JSON can handle) plus zipping (dictionary coding) takes care of, what, 99.9999% of the issues folks have with one type or the other?

Re: Internet Object – A JSON alternative data serialization format

#17
I've been looking at data serialisation formats recently.

- JSON - TOML - CSON - INI - ENO - XML

I like CSV for tabular data obviously. This looks, as others have mentioned, like CSV with better metadata.

I like INI for its simplicity. JSON is good for more complicated data, but I have to say I like CSON.

Re: Internet Object – A JSON alternative data serialization format

#18
post #13

Since strings don't need to be quoted, what happens during deserialization if you want the string "T"? Does this lead to the equivalent of the Norway-Problem of YAML [0]? Is the space between the key and the type necessary? If not, how to distinguish between objects and types? Does the validation offer some form of unions or mutual exclusion? [0]: https://hitchdev.com/strictyaml/why/implicit-typing-removed/

YAML and its "Arrays" are really broken. The problem I see with Internet Object is that it's also implying this kind of mechanism.

Every time I read about new formats, they seem to get either the 1-n relations or the n-n relations implemented well, but not both. I guess that's what's so hard about map/reduce...

Regarding YAML: somebody on HN mentioned his project DIXY a couple years ago, and it's much much _much_ easier to parse than YAML. [1] I'm using this over YAML pretty much everywhere now.

[1] https://github.com/kuyawa/Dixy

Re: Internet Object – A JSON alternative data serialization format

#19
Json is a good format to represent results of aggregation queries (group by in sql) using nesting and storing data in a single file.

Without that you would need to either

  1. store multiple not-nested (tabular, eg. csv) files and join them at the time of use.
  2. denormalize all these csvs into a single big csv duplicating the same values over and over. Compression should handle this at storage time, bht you still pay the cost when reading.
  3. store values by columns, not by rows, adding various RLE and dict encodings to compress repeated values in columns, making the files not human friendly
  4. once you store it in columns and make it unreadable, just store it as binary instead of text. You get parquet
Json and csb are simple and for that reason they won and will stay with us no matter how hard you try to add features to it.

That said I think adding a trailing comma and comments to json wouldn't be a big stretch.

The battle will be for the best columnar binary format. Parquet is the closest to a standard, but it seems to be used only as a standard for a storage. Big data systems still uncompress it and work with their own representation. The holy grail is when you get a columnar format which is good enough that big data systems use it as their underlying data representation instead of coming up with their own. I suspect such format will come from something like open sourced Snowflake, Clickhouse, Chaossearch or something like that, which has battle tested performant algorithms on them, instead of designed by committee, such as parquet.

Post reply on HN