Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Object – A JSON alternative data serialization format

internetobject.org

Post reply on HN
Print view View original

61–70 of 83 posts

Re: Internet Object – A JSON alternative data serialization format

#61

Post by snidane »

throwaway81523 wrote: post #26
snidane wrote: post #19

Json is a good format to represent results of aggregation queries (group by in sql) using nesting and storing data in a single file. Without that you would need to either 1. store multiple not-nested (tabular, eg. csv) files and join them at the time of use. 2. denormalize all these csvs into a single big csv duplicating the same values over and over. Compression should handle this at storage time, bht you still pay…

> That said I think adding a trailing comma and comments to json wouldn't be a big stretch. Sadly, json's designers suffered from the same hubris as the designers of markdown and gemini, when they decided to not include a version number in the file format. So you are kind of hosed if you want to make a change like that. Before json there was xml (ugh), but before xml there were Lisp S-expressions, which seem to have…

It's just a matter of parser implementation. These changes are backwards compatible. If python decided to add support for comments and trailing commas in json.loads, that would become the new standard, at least for data scientists, not for web devs. All the other ones would then follow.

Re: Internet Object – A JSON alternative data serialization format

#62

Post by SV_BubbleTime »

mofosyne wrote: post #52

Hope it supports semantic tagging like in CBOR

Are you using your own tags in CBOR? What is the use case?

I figured that because I need to describe the tag, it was just as easy to not use tags and describes the elements that would make one up.

Re: Internet Object – A JSON alternative data serialization format

#63

Post by snidane »

liuliu wrote: post #25
snidane wrote: post #19

Json is a good format to represent results of aggregation queries (group by in sql) using nesting and storing data in a single file. Without that you would need to either 1. store multiple not-nested (tabular, eg. csv) files and join them at the time of use. 2. denormalize all these csvs into a single big csv duplicating the same values over and over. Compression should handle this at storage time, bht you still pay…

You mean, Apache Arrow?

Partially.

The problem with Apache Arrow and Parquet is that you have two - one for storage and one for computation - but in the end you only want one for both. You want to run fast algorithms on memory mapped compressed columns. Not doing this stupid deserialization from parquet to arrow.

Parquet and arrow are designed by committee and try to accomplish too much for that matter. While that's good for some cases, my prediction is that there will exist a data processing system in the future whose file format will support that and be good enoigh for most data intensive applications. It will not be feature complete, like json, but will be good enough. Some devs from then on will complain about adding this and that feature to that format, but majority will be happy as they are now with json. Such format can only come from industry, not from a committee.

Re: Internet Object – A JSON alternative data serialization format

#65

Post by throwaway81523 »

snidane wrote: post #61
throwaway81523 wrote: post #26

Earlier quoted context omitted.

> That said I think adding a trailing comma and comments to json wouldn't be a big stretch. Sadly, json's designers suffered from the same hubris as the designers of markdown and gemini, when they decided to not include a version number in the file format. So you are kind of hosed if you want to make a change like that. Before json there was xml (ugh), but before xml there were Lisp S-expressions, which seem to have…

It's just a matter of parser implementation. These changes are backwards compatible. If python decided to add support for comments and trailing commas in json.loads, that would become the new standard, at least for data scientists, not for web devs. All the other ones would then follow.

Now whatever generates your data has to know what parser is going to read the data. The parser can't tell right away whether the data has those trailing commas. They are optional, so they might not start appearing until after gigabytes of output have gone by. So you can't count on a quick error message in the event of a version mismatch.

Re: Internet Object – A JSON alternative data serialization format

#66

Post by charles_f »

a) why would you want to remove the field names, this is making it so much harder to debug and very brittle, since now you're dependent on the order of fields. No mention of how you handle versioning as well. Back to csv

> However, this time, something felt wrong; I realized that with the JSON, we were exchanging a huge amount of unnecessary information to and from the server

b) Text size really ain't an issue given that we're talking about typically just a few kb on gzipped protocols over hundreds of mbps connections. Compactness sounds like a bad argument to me.

c) "json doesn't have schema built in is a really dubious argument". If you want schemas you can still get them using json-schema, and if you don't you can still understand the message using the field names, which makes for a degraded schema ; which doesn't exist in the case of internet objects. If you don't have the schema, go figure what's in there

What really gives it to me is the comparison at the bottom between internet objects anf json; json looks better to me.

Looks like it's an idea executed over a bad premise

Re: Internet Object – A JSON alternative data serialization format

#67

Post by dang »

A couple small past threads:

JSON Alternative – Internet Object - https://news.ycombinator.com/item?id=21220405 - Oct 2019 (12 comments)

Show HN: Internet Object – a thin, robust and schema oriented JSON alternative - https://news.ycombinator.com/item?id=20982180 - Sept 2019 (8 comments)

Re: Internet Object – A JSON alternative data serialization format

#68

Post by BiteCode_dev »

cookiengineer wrote: post #18
nikeee wrote: post #13

Since strings don't need to be quoted, what happens during deserialization if you want the string "T"? Does this lead to the equivalent of the Norway-Problem of YAML [0]? Is the space between the key and the type necessary? If not, how to distinguish between objects and types? Does the validation offer some form of unions or mutual exclusion? [0]: https://hitchdev.com/strictyaml/why/implicit-typing-removed/

YAML and its "Arrays" are really broken. The problem I see with Internet Object is that it's also implying this kind of mechanism. Every time I read about new formats, they seem to get either the 1-n relations or the n-n relations implemented well, but not both. I guess that's what's so hard about map/reduce... Regarding YAML: somebody on HN mentioned his project DIXY a couple years ago, and it's much much _much_ eas…

Yaml has so many problems. Python 3.10 raised a new one to my attention when the core devs realized their arrays of versions contained twice 3.1 and no 3.10. Indeed, if write unquotted ascii, yaml gives you strings. Except if it can cast it to a number that is.

TOML is better, but it still has more gotchas that necessary. So much I find it easier to just edit a python file

I'm thinking of giving a try to cue. Any feedback ?

Re: Internet Object – A JSON alternative data serialization format

#69

Post by liuliu »

snidane wrote: post #63
liuliu wrote: post #25

Earlier quoted context omitted.

You mean, Apache Arrow?

Partially. The problem with Apache Arrow and Parquet is that you have two - one for storage and one for computation - but in the end you only want one for both. You want to run fast algorithms on memory mapped compressed columns. Not doing this stupid deserialization from parquet to arrow. Parquet and arrow are designed by committee and try to accomplish too much for that matter. While that's good for some cases, my…

Right. That's why I am more interested in arrow than parquet. Going from a pure compressed storage format to incorporate computation would be more difficult than going from memory-mapped / computation format to long-term storage. Arrow already made some good choices regarding data exchange over wire, these are translatable to data exchange over time.

Of course, I am only dealing with a few hundreds GiB data, not sure at larger scale whether arrow fails.

Re: Internet Object – A JSON alternative data serialization format

#70

Post by irq-1 »

cookiengineer wrote: post #18
nikeee wrote: post #13

Since strings don't need to be quoted, what happens during deserialization if you want the string "T"? Does this lead to the equivalent of the Norway-Problem of YAML [0]? Is the space between the key and the type necessary? If not, how to distinguish between objects and types? Does the validation offer some form of unions or mutual exclusion? [0]: https://hitchdev.com/strictyaml/why/implicit-typing-removed/

YAML and its "Arrays" are really broken. The problem I see with Internet Object is that it's also implying this kind of mechanism. Every time I read about new formats, they seem to get either the 1-n relations or the n-n relations implemented well, but not both. I guess that's what's so hard about map/reduce... Regarding YAML: somebody on HN mentioned his project DIXY a couple years ago, and it's much much _much_ eas…

Dixy looks easy, but "There is only one simple rule. In Dixy, everything is a dictionary [string:string]" isn't accurate or helpful.

It's also [string:dictionary] and [string:?] where ? means nil. White space matters, and tab is fixed at 4 spaces wide. When creating text from a dictionary it adds "# Dixy 1.0\n\n" which means loading and saving will change the file every time! Not sure what other issues there are, but I noticed this line:

    // TODO: if key is numeric, parse as Array
It does look simple though. It'd be nice if someone made strict rules and addressed the corner cases.
Post reply on HN