Show HN: Catj – A new way to display JSON files • Hacker News Board

Re: Show HN: Catj – A new way to display JSON files

#121

Post by jolmg » Mon, Jun 24, 2019, 3:42 PM UTC

sethish wrote: post #119

jolmg wrote: post #36

Earlier quoted context omitted.

You're right. Good tip: jq -r ' tostream | select(length > 1) | ( .[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ) + " = " + (.[1] | @json) ' EDIT: For those who want to save this script, you can put just the jq code in an executable file with the shebang: #!/usr/bin/jq -rf

This would be more suitable to large json files if used with the `--stream` flag. Here's my take on it: jq -c --stream ' . as $in | select(length == 2) | ( $in[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | add ) + " = " + ($in[1] | tostring)' Using `--stream` allows jq to start before parsing the entire json file. In my experience, a 700mb json file can take up 5gb of ram in either jq o…

You're right about `--stream`, but you didn't need the variable assignment. Also, `-c`, besides the fact that it's not available in jq-1.5 which some people are using, is very pointless in this situation, since we're looking to output text, and `-c` is for outputting objects/arrays in a compact format. The fact that `-r` wasn't used causes jq to output the text encoded as json strings. So, instead of outputting:

  .movie.name = "Interstellar"
  .movie.year = 2014
  .movie.is_released = true
  .movie.else = "Christopher Nolan"
  .movie.cast[0] = "Matthew McConaughey"
  .movie.cast[1] = "Anne Hathaway"
  .movie.cast[2] = "Jessica Chastain"
  .movie.cast[3] = "Bill Irwin"
  .movie.cast[4] = "Ellen \\\\ Burstyn"
  .movie.cast[5] = "Michael Caine"

You're outputting:

  ".movie.name = Interstellar"
  ".movie.year = 2014"
  ".movie.is_released = true"
  ".movie.else = Christopher Nolan"
  ".movie.cast[0] = Matthew McConaughey"
  ".movie.cast[1] = Anne Hathaway"
  ".movie.cast[2] = Jessica Chastain"
  ".movie.cast[3] = Bill Irwin"
  ".movie.cast[4] = Ellen \\\\ Burstyn"
  ".movie.cast[5] = Michael Caine"

Another point is how the strings at the right of the `=` are displayed. They should be quoted. The reason why they're not is because you piped the second element to `tostring` instead of `@json`.

A better version of your suggestion would've been:

  jq -r --stream '   
    select(length > 1)  
    | (
      .[0] | map(
        if type == "number"
        then "[" + tostring + "]"
        else "." + .
        end
      ) | add
    ) + " = " + (.[1] | @json)
  '

The use of `length > 1` instead of `length == 2` is a minor point, but if a future version jq decides to sometimes put 3 elements in these arrays, your filter would ignore those when we're likely to also want those. `length > 1` ensures what we need, that there are at least the elements that we're going to be using, while `length == 2` might filter some of those out, even if it's not right now.

Your use of `add` is neat, though. I wouldn't have thought of that.

Re: Show HN: Catj – A new way to display JSON files

#122

Post by alinspired » Mon, Jun 24, 2019, 5:59 PM UTC

jolmg wrote: post #20

I was curious if this could be doable with jq, and apparently it is: jq -j ' [ [ paths(scalars) | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ], [ .. | select(scalars) | @json ] ] | transpose | map(join(" = ") + "\n") | join("") ' EDIT: Got the string quoting and escaping. EDIT 2: For those who want to save this script, you can put just the jq code in an executable file with the s…

Following https://github.com/stedolan/jq/issues/243 i commonly use https://github.com/joelpurra/har-dulcify/blob/master/src/uti... to explore unfamiliar json, ie:

  $ docker inspect 620f55df9177| structure.sh |grep -i addr
   .[].NetworkSettings.GlobalIPv6Address
   .[].NetworkSettings.IPAddress
   .[].NetworkSettings.LinkLocalIPv6Address
   .[].NetworkSettings.MacAddress
   .[].NetworkSettings.Networks.bridge.GlobalIPv6Address
   .[].NetworkSettings.Networks.bridge.IPAddress
   .[].NetworkSettings.Networks.bridge.MacAddress
  
  $ docker inspect 620f55df9177| jq .[].NetworkSettings.IPAddress
   "192.168.0.2"

Re: Show HN: Catj – A new way to display JSON files

#123

Post by alinspired » Mon, Jun 24, 2019, 7:58 PM UTC

waveforms wrote: post #113

kfrzcode wrote: post #88

Earlier quoted context omitted.

jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at , line 3: jq -j ' jq: 1 compile error This, uh, doesn't work for me on jq-1.5.1.

#!/usr/local/bin/jq -rf tostream | select(length > 1) | ( .[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ) + " = " + (.[1] | @json)

i need to understand how #! works, ie `#!/usr/bin/jq --stream -rf" errors with `/usr/bin/jq: Unknown option --stream -rf`

`#!/usr/bin/jq -rf ` with tostream wrapper in code works fine

Re: Show HN: Catj – A new way to display JSON files

#124

Post by jolmg » Tue, Jun 25, 2019, 1:03 AM UTC

alinspired wrote: post #123

waveforms wrote: post #113

Earlier quoted context omitted.

#!/usr/local/bin/jq -rf tostream | select(length > 1) | ( .[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ) + " = " + (.[1] | @json)

i need to understand how #! works, ie `#!/usr/bin/jq --stream -rf" errors with `/usr/bin/jq: Unknown option --stream -rf` `#!/usr/bin/jq -rf ` with tostream wrapper in code works fine

Like another thread mentioned, shebang (#!) parsing is non-standard. In macOS, I think what you tried would work like you'd expect, but it'd work differently on linux. The reason is that in linux, after parsing the path to the executable and a space, everything else is taken as a single argument. So if you were in bash, what you did would be the equivalent of doing:

  jq "--stream -rf" path/to/script

and jq doesn't know of any one option called "--stream -rf".

I haven't seen the discussions around these design decisions in the different OSes, but I imagine the crux of the matter is that you have to pick somewhere to stop, and where you chose to stop is largely arbitrary.

I mean, you can have the OS interpret shebangs with multiple arguments, but then you'll want to be able to put spaces in these arguments, so you'll want quoting, and then you'll want to put special characters like newlines inside, so you'll want escaping, etc.

The OS can implement all these things in execve()'s logic, but it might also be preferable to keep the logic simple in the interest of avoiding security-harming bugs. You know, less code, less bugs, less vulnerabilities.

If --stream had a single letter option equivalent, you could stick it together with the other ones. However, since it doesn't, your only option to make a portable script is to use a shell shebang like #!/bin/bash, and then do:

  exec jq --stream -rf ...

You might feel that this single argument restriction sucks and is definitely inferior to any implementation of multiple argument shebangs. I don't know if macOS shebangs support quoting, but if they don't and simply split on spaces, then I can tell you they can't do hacky stuff like writing code in a shebang like this:

> https://unix.stackexchange.com/questions/365436/choose-inter...

Granted, it's bad practice, but a little cool nevertheless.

Re: Show HN: Catj – A new way to display JSON files

#125

Post by deostroll » Wed, Jun 26, 2019, 6:40 AM UTC

Sweet! I find this tool extremely useful for parsing dmn files in javascript (via xml2js). But right now I am more interested in a similar feature for xml/html documents. Hoping if someone can point me to it. Thanks.

Re: Show HN: Catj – A new way to display JSON files

#126

Post by mkesper » Sat, Jun 29, 2019, 6:17 AM UTC

tedivm wrote: post #110

ficklepickle wrote: post #34

Earlier quoted context omitted.

What an interesting idea! On my last contract, I had to deal with 100mb+ JSON responses from a barely-documented API. This would have come in handy when I was figuring it out. I've used JSONExplorer for this purpose, but it is web based and doesn't handle files this large. Extending the filesystem metaphor to JSON data and re-using the same commands strikes me as a great idea. Did another project inspire you, or did…

I came up with this idea on my own when dealing with the AWS Bulk API files. I did make a "Show HN" but it got a total of four upvotes.

HN is very sensitive to timing, afaik the moderators allow resubmissions when that seems to be the case.

Re: Show HN: Catj – A new way to display JSON files

#127

Post by facorreia » Sat, Jun 29, 2019, 11:33 PM UTC

jolmg wrote: post #121

sethish wrote: post #119

Earlier quoted context omitted.

This would be more suitable to large json files if used with the `--stream` flag. Here's my take on it: jq -c --stream ' . as $in | select(length == 2) | ( $in[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | add ) + " = " + ($in[1] | tostring)' Using `--stream` allows jq to start before parsing the entire json file. In my experience, a 700mb json file can take up 5gb of ram in either jq o…

You're right about `--stream`, but you didn't need the variable assignment. Also, `-c`, besides the fact that it's not available in jq-1.5 which some people are using, is very pointless in this situation, since we're looking to output text, and `-c` is for outputting objects/arrays in a compact format. The fact that `-r` wasn't used causes jq to output the text encoded as json strings. So, instead of outputting: .mov…

Thanks! I've made a gist out of it:

https://gist.github.com/fernandoacorreia/4b67a41bbe227654868...