Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JC – JSONifies the output of many CLI tools

kellyjonbrazil.github.io

Post reply on HN
Print view View original

81–90 of 136 posts

Re: JC – JSONifies the output of many CLI tools

#81

Post by pessimizer »

I don't know why I never considered this sort of option to powershell out bash. My only problem with it is that it's in python (also why I don't really use jq), and that it's not something that just sets aliases behind the scenes.

If this were written in a performant language, if it simply aliased (i.e. invisibly) all common cli commands to a wrapper which would obviate the need for all of the text processing between steps in command pipelines, if it were versioned and I could include the version number in scripts, and finally if I could run versioned scripts through it to compile them into standard bash scripts (a big ask), I'd give it a 3 month test starting today. There'd be nothing to lose.

Just putting that out there for people who like to rewrite things in Rust. A slightly different version of this concept could allow for nearly friction-free adoption.

Re: JC – JSONifies the output of many CLI tools

#82

Post by hinkley »

I discovered process.send() in Node a couple years ago and it made the decision to fork a child process a lot easier. No need to sanitize command line output when you can do direct IPC over a connection that uses JSON under the hood.

The itch I can’t seem to scratch is how to run tasks in parallel and have logs that are legible to coworkers. We do JSON formatted logs in production and I’m wondering if something like this would help solve that set of problems.

Re: JC – JSONifies the output of many CLI tools

#83

Post by cstrahan »

pessimizer wrote: post #81

I don't know why I never considered this sort of option to powershell out bash. My only problem with it is that it's in python (also why I don't really use jq), and that it's not something that just sets aliases behind the scenes. If this were written in a performant language, if it simply aliased (i.e. invisibly) all common cli commands to a wrapper which would obviate the need for all of the text processing between…

> also why I don't really use jq)

You also don’t like software written in C (the language jq is written in)?

Re: JC – JSONifies the output of many CLI tools

#84

Post by simonkagedal »

slightwinder wrote: post #43
rob74 wrote: post #14

Great idea, but sounds like a maintenance nightmare to me. Not only that many users will complain that their favorite CLI tool isn't supported, but also a new release of any of the supported CLI tools might break the support without any kind of warning, as I don't think changes to the (human-readable) output are considered major changes.

It doesn't even need a new release. jc can already fail because of details like the system-language. With my local language, on a simple output of ls -l, it's parsing {"filename":"drwxr-xr-x 16 root root 4096 Oct 4 11:21 ."} instead of {"filename":".","flags":"drwxr-xr-x","links":16,"owner":"root","group":"root","size":4096,"date":"Oct 4 11:21"} with LANG=US. This makes it really hard to trust such a tool.

This is true. However, the issue is with the tool outputting unstructured data to begin with. As the `jc` author says, the best would be if tools supported structured output formats to begin with.

Any custom parser of ls output would potentially have the same problem. Of course, it can be improved though – for example by looking at LANG – and it would be nice for such improvements to get into `jc`, so that other tools can rely on it at least more than doing the parsing directly themselves.

Re: JC – JSONifies the output of many CLI tools

#85

Post by lstamour »

anonymoushn wrote: post #79
dotancohen wrote: post #78

Earlier quoted context omitted.

WC has a "character" flag that really just counts bytes: $ echo דותן | wc -c 9 Note that each letter is two bytes, and the newline is an additional byte. So you could pipe (or tree) to wc to count bytes. For a hypothetical stdmeta on fd 3, that might look like this (piping stdout to devnull): $ foo 3>&1 > /dev/null | wc -c

I'm not sure what you are answering. I am writing about the thing where some programs intentionally write different byte streams to stderr and stdout that will be interleaved a certain way if stdout and stderr turn out the be the same thing, but if the user wishes to distinguish which bytes were written to which stream, they can no longer recover the correct interleaving of the bytes.

The best compromise I can think of would be to prepend the line source to the combined output, so lines in the combined output can be distinguished. But depending on how you do that, you might have timing issues. In fact, any form of interleaving the outputs of stderr and stdout are technically subject to timing issues because even the source application can’t specify that bytes in one stream should follow bytes in another stream. https://stackoverflow.com/questions/12517519/how-to-redirect... talks a bit about the problem, and I’m sure someone somewhere has written a general-purpose tool, but there are timing edge cases here and often the solution is seen as keeping errors messages exclusively in stdout (or logs) or more practically, not worrying about preserving order and writing errors exclusively to stderr without worrying as much about strictly when errors interleave. If you add timestamps to your line outputs, or add some form of line numbers to the output, you can restore order after logging but it would be application-specific to do so. From a practical perspective, listening to both streams and quickly appending to a list with a data structure containing the source and message text is probably your closest approximation if writing code to handle output streams and you want an combined output that can still source which stream came from which source. (Again subject to timing errors)

Re: JC – JSONifies the output of many CLI tools

#86

Post by sbt567 »

dmoura wrote: post #31

This is great! I am the author of SPyQL [1]. Combining JC with SPyQL you can easily query the json output and run python commands on top of it from the command-line :-) You can do aggregations and so forth in a much simpler and intuitive way than with jq. I just wrote a blogpost [2] that illustrates it. It is more focused on CSV, but the commands would be the same if you were working with JSON. [1] https://github.com…

Wow, this looks super useful. Will definitely check this out. Thanks

Re: JC – JSONifies the output of many CLI tools

#87

Post by freedomben »

pessimizer wrote: post #81

I don't know why I never considered this sort of option to powershell out bash. My only problem with it is that it's in python (also why I don't really use jq), and that it's not something that just sets aliases behind the scenes. If this were written in a performant language, if it simply aliased (i.e. invisibly) all common cli commands to a wrapper which would obviate the need for all of the text processing between…

Why is the python an issue for you? I also dislike and avoid tools written in python, unless they have an rpm available. If there's a package then I don't so much care what language it's in.

Re: JC – JSONifies the output of many CLI tools

#88

Post by maxbond »

synergy20 wrote: post #42

to make this great tool truly universal, it has to be written in c instead of python these days, then provide python|javascript|etc bindings if possible. I'd like to use it on embedded systems, where python is too large to fit. this tool can be widely deployed just like awk|sed|etc but it has to be in C for that.

Seems to me like anywhere Python is too bulky to work, a layer serializing and deserializing JSON at every pipeline step is likely to exhaust your memory too.

Re: JC – JSONifies the output of many CLI tools

#89

Post by psychoslave »

codedokode wrote: post #47

I think that JSON is a bad choice here. It is obvious that CLI commands should produce machine-readable output because they are often used in scripts, and accept machine-readable input as well. Using arbitrary text output was a mistake because it is difficult to parse, especially when spaces and non-ASCII characters are present. A good choice would be a format that is easily parsed by programs but still readable by t…

Following the project documentation, you easily come to:

  jc dig example.com | jq       
  [
    {
      "id": 30081,
      "opcode": "QUERY",
      "status": "NOERROR",
      "flags": [
        "qr",
        "rd",
        "ra"
      ],
      "query_num": 1,
      "answer_num": 1,
      "authority_num": 0,
      "additional_num": 1,
      "opt_pseudosection": {
        "edns": {
          "version": 0,
          "flags": [],
          "udp": 4096
        }
      },
      "question": {
        "name": "example.com.",
        "class": "IN",
        "type": "A"
      },
      "answer": [
        {
          "name": "example.com.",
          "class": "IN",
          "type": "A",
          "ttl": 56151,
          "data": "93.184.216.34"
        }
      ],
      "query_time": 0,
      "server": "192.168.1.254#53(192.168.1.254)",
      "when": "Thu Nov 03 14:06:40 CET 2022",
      "rcvd": 56,
      "when_epoch": 1667480800,
      "when_epoch_utc": null
    }
  ]
Rather readable to my mind. And you can rather easily transform it to your preferred human readable output format I guess.

Re: JC – JSONifies the output of many CLI tools

#90

Post by runlevel1 »

dotancohen wrote: post #63
asicsp wrote: post #11

See also: * "Bringing the Unix philosophy to the 21st century (2019)" ( https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-ph... ) - https://news.ycombinator.com/item?id=28266193 238 points | Aug 22, 2021 | 146 comments * "Tips on adding JSON output to your CLI app" ( https://blog.kellybrazil.com/2021/12/03/tips-on-adding-json-... ) - https://news.ycombinator.com/item?id=29435786 183 points | 11 months ago | 1…

If you're bringing that up, then this is the place to spread the word about my dream of a stdmeta file descriptor. https://unix.stackexchange.com/questions/197809/propose-addi... Just like we have stdout and stderr, header lines such as those produced by `ps` should be printed to stdmeta. Curl is the worse offender here, outputing meta lines to stderr instead of stdout. A stdmeta file descriptor would make it clear w…

That reminds me of something I've wanted for quite a while:

A ringbuffer filetype. Similar to a named pipe file (see: fifo(7)[^1]), but without consuming the contents on read and automatically rotating out the oldest lines.

Of course, there would be some complexities around handling read position as lines are being rotated out from under you.

[1]: https://linux.die.net/man/7/fifo

Post reply on HN