Ask HN: How do you view large JSON files?

Ha! It is funny because it is true. I remember listning to one of Joe Armstrong's talks (React 2014 conf and he talks at some point ( https://youtu.be/rQIE22e0cW8?t=2009 ) how parsing is rather expensive CPU-wise and also bandwidth expensive. Especially in mobile networks. The company he works for control data paths for smart phones to the internet, and they sweat wasting every little bit because it eats into the precious bandwitdth available to consumers -- and what do developers do? -- they shove JSON through that channel in the application level!

It was a silly observation but it is also true at some level. JSON might be easy to read, but reading a 100M json still needs a special editor.

Another funny observation Joe made at some point when a response to him calling JSON out, because "after all, you can see JSON" was that, your eyes cannot see JSON, they see photons bouncing from the screen. You still use an editor or some other translation program to display it and read it. So at some point might as well use binary (thrift, protobufs, sqlite, ...).

I have a question related to this which nobody has really touched on yet.

Imagine you have a json structure like this:

{"records" : [ {"name":"joe", ...lots of other fields in each record}, {"name":"fred", ...lots of other fields}, ... thousands of records ]}

Now a lot of people in this thread have mentioned tools for collapsing the json, but the trouble I have with this is that you can't browse the record names without opening each one and looking at all the fields. It would be nice if there was a tool that took a few key identifiers (name, id) and "bubbled them up" so that in a collapsed view you would see something like:

{"records":[ {...click to expand...}, // name=joe {...click to expand...}, // name=fred ]}

I have the same problem when working with XML as well (though in XML sometimes the ID's are attributes of the parent which mitigates the problem, but other times the ID would be nested in a child element of the structure more like you would have in json). I even found it to be enough of a problem that I wrote my own XML editor to solve this issue.

Of course one issue with this is that there's so clear standard as to which fields of an object represent "ID" information which would be important/useful to "bubble up" to the next level when collapsing. It would have to be something user-configurable (though having some sensible defaults like looking for "name" and "id" keys would work in a lot of cases). In the XML world, there's probably something to do with Schemas that would help with this problem, and fancy editors which understand your data using a schema, though some of the editors I looked at which went into that level of detail seemed like way overkill for what i wanted to do.

So essentially my question here is whether this concept of "collapse child, but keep important identifying information of the collapsed child visible" exists in any json tools? is this a thing that has a name/buzzword associated with it that I don't know about? or is it purely an issue that's my own personal quirk which nobody else cares about?

A human can't easily understand the structure of a large JSON file - JSON is internally self-describing, so there is no guarantee that the sample of records currently visible in your editor's viewport are representative of the whole.

Instead, use a tool like Schema Guru (https://github.com/snowplow/schema-guru ; disclaimer: we wrote this at Snowplow) to programmatically extract the JSON Schema (http://json-schema.org/) which represents all JSON instances in the file.

I wrote a tool some time ago. It's not a viewer. It just shows the structure. https://github.com/ilyash/show-struct

On the macOS I've had some success with "Cocoa JSON Editor.app". It takes a while to load files, but they are presented in an outline view you can search and browse.