Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JC – JSONifies the output of many CLI tools

kellyjonbrazil.github.io

Post reply on HN
Print view View original

111–120 of 136 posts

Re: JC – JSONifies the output of many CLI tools

#111

Post by naikrovek »

This seems very fragile, to me, without support from the application whose output is converted to JSON.

minor updates to command-line tools can and do subtly alter the textual output of the tool, and the outputs of these tools are not standardized.

This is a step towards "objects passing messages" as originally conceived by Alan Kay, if my incomplete understanding of what he's said is correct, and that's a good thing, I think. Objects passing messages around is a very solid model for computing, to me. Note that I am stupid and don't understand much, if I'm honest.

Re: JC – JSONifies the output of many CLI tools

#112

Post by kangalioo »

tazjin wrote: post #16
vesinisa wrote: post #15

Seems to be part of the Arch repository: https://archlinux.org/packages/community/any/jc/ Why does this site recommend using "paru", "aura" or "yay" to install it on Arch? I have been using Arch for a decade or so but have never even heard of such tools. They don't even have pages in the Arch wiki, and only yay ("Pacman wrapper and AUR helper written in go") is available via the standard repository. Begs the question…

It's a program to wrap stable CLI programs, makes sense that they recommend a wrapper around a stable program like pacman

Is this a snarky comment implying that the CLI programs and pacman are better left unwrapped?

Re: JC – JSONifies the output of many CLI tools

#113

Post by zokier »

WorldMaker wrote: post #100
codedokode wrote: post #47

I think that JSON is a bad choice here. It is obvious that CLI commands should produce machine-readable output because they are often used in scripts, and accept machine-readable input as well. Using arbitrary text output was a mistake because it is difficult to parse, especially when spaces and non-ASCII characters are present. A good choice would be a format that is easily parsed by programs but still readable by t…

> A good choice would be a format that is easily parsed by programs but still readable by the user. I think the powershell approach is a good one here too: powershell commands output binary streams of objects rather than text and it is powershell itself that has several standard ways of human readable outputs, most of which are automatic (but easily tweaked with an extra pipe or two). Standard human readable forms ar…

The thing to understand with PowerShell is that the way it pipelines objects is enabled by the fact that it is all happening in-process within one .net runtime. It is significantly more difficult to achieve anything similar with several independent processes being piped together

Re: JC – JSONifies the output of many CLI tools

#114

Post by kbrazil »

vincnetas wrote: post #72
exabrial wrote: post #71

Have a parseable output is great. What would be more incredible is to have a parseable output with a schema definition and/or formal grammar of some sort.

Schemas are provided on linked page near definitions of parsers.

You can also run `jc -h --dig` to get the parser details that include the schema.

Having true JSON Schema[0] is being considered, but on the back-burner due to the sheer number of parsers to build schemas for. Also, it is more difficult to accurately define the schema for a small subset of parsers since their command output are so variable.

[0] https://json-schema.org/

Re: JC – JSONifies the output of many CLI tools

#115

Post by synergy20 »

bitwize wrote: post #106
synergy20 wrote: post #104

Earlier quoted context omitted.

on embedded rust is still much larger than c because rust links its stdlib statically.

There is such a thing as no-stdlib Rust deployments.

which is for MCUs only and is like a bare metal language without any libraries, its use is very limited.

all I need is that rust's stdlib can be linked dynamically just like c/c++/java/whatever, if that happens I'm ready to switch to it.

Re: JC – JSONifies the output of many CLI tools

#116

Post by kbrazil »

slightwinder wrote: post #43
rob74 wrote: post #14

Great idea, but sounds like a maintenance nightmare to me. Not only that many users will complain that their favorite CLI tool isn't supported, but also a new release of any of the supported CLI tools might break the support without any kind of warning, as I don't think changes to the (human-readable) output are considered major changes.

It doesn't even need a new release. jc can already fail because of details like the system-language. With my local language, on a simple output of ls -l, it's parsing {"filename":"drwxr-xr-x 16 root root 4096 Oct 4 11:21 ."} instead of {"filename":".","flags":"drwxr-xr-x","links":16,"owner":"root","group":"root","size":4096,"date":"Oct 4 11:21"} with LANG=US. This makes it really hard to trust such a tool.

Hi there! `jc` author here. Yes, it is a documented caveat[0] that the `C` or `en_US.UTF-8` locales should be used for best results.

It's not unheard of for tools to require `C` locale for proper parsing:

    $ LC_ALL=C ls -l | jc --ls
This is one of many inherent issues with using unstructured text as an API. That's why I believe there should be a JSON (or at least some other widely used format[1]) option for tools that have output that would be useful in scripts.

[0] https://github.com/kellyjonbrazil/jc#locale

[1] formats should have good library support across many languages and nice filter/query capabilities from the command-line

Re: JC – JSONifies the output of many CLI tools

#117

Post by WorldMaker »

zokier wrote: post #113
WorldMaker wrote: post #100

Earlier quoted context omitted.

> A good choice would be a format that is easily parsed by programs but still readable by the user. I think the powershell approach is a good one here too: powershell commands output binary streams of objects rather than text and it is powershell itself that has several standard ways of human readable outputs, most of which are automatic (but easily tweaked with an extra pipe or two). Standard human readable forms ar…

The thing to understand with PowerShell is that the way it pipelines objects is enabled by the fact that it is all happening in-process within one .net runtime. It is significantly more difficult to achieve anything similar with several independent processes being piped together

Well, yes, powershell takes some shortcuts and has the advantage that .NET has a strong object system.

If you were to build it from scratch with the idea of "shared nothing" applications similar to the unix model with text files, it's not that much more difficult with just about any sort of object or message broker. You could easily imagine a world with a dbus based "REPL"/shell, for instance. Or a different approach easily imaginable if you still want to focus on unix-style streams/files between processes would be something like BSON streams (thought it would still have some serialization/deserialization overhead).

Re: JC – JSONifies the output of many CLI tools

#118

Post by DonHopkins »

WorldMaker wrote: post #100
codedokode wrote: post #47

I think that JSON is a bad choice here. It is obvious that CLI commands should produce machine-readable output because they are often used in scripts, and accept machine-readable input as well. Using arbitrary text output was a mistake because it is difficult to parse, especially when spaces and non-ASCII characters are present. A good choice would be a format that is easily parsed by programs but still readable by t…

> A good choice would be a format that is easily parsed by programs but still readable by the user. I think the powershell approach is a good one here too: powershell commands output binary streams of objects rather than text and it is powershell itself that has several standard ways of human readable outputs, most of which are automatic (but easily tweaked with an extra pipe or two). Standard human readable forms ar…

Not binary streams of serialized objects, but arrays of pointers to live objects in one process's memory. No serialization / deserialization, binary or text, or piping between processes, just passing pointers to live objects between cmdlets in the same address space. That's quite different and vastly more efficient than serializing and deserializing text or binary data between every step in different processes connected by pipes.

https://en.wikipedia.org/wiki/PowerShell#Pipeline

>As with Unix pipelines, PowerShell pipelines can construct complex commands, using the | operator to connect stages. However, the PowerShell pipeline differs from Unix pipelines in that stages execute within the PowerShell runtime rather than as a set of processes coordinated by the operating system. Additionally, structured .NET objects, rather than byte streams, are passed from one stage to the next. Using objects and executing stages within the PowerShell runtime eliminates the need to serialize data structures, or to extract them by explicitly parsing text output. An object can also encapsulate certain functions that work on the contained data, which become available to the recipient command for use. For the last cmdlet in a pipeline, PowerShell automatically pipes its output object to the Out-Default cmdlet, which transforms the objects into a stream of format objects and then renders those to the screen.

Re: JC – JSONifies the output of many CLI tools

#119

Post by UncleEntity »

exabrial wrote: post #71

Have a parseable output is great. What would be more incredible is to have a parseable output with a schema definition and/or formal grammar of some sort.

> What would be more incredible is to have a parseable output with a schema definition and/or formal grammar of some sort.

I’ve been trying to build something like this but simply don’t have the free time currently.

The plan: adapt the parser VM from lpeg (or similar, there’s a paper I’ve been reading on an Earley parser VM) into a command line app that takes a grammar + text input (or stdin) and spits out json to a file (or stdout). Probably not as general purpose as this one but also wouldn’t need a pull request to add a new format.

All the pieces are there but without the free time…

I was actually curious if there was any demand for such a thing, I just want it to parse my payroll statements because this billion dollar company can only manage crappy pdfs and, well, it’s an interesting problem.

—edit—

Oh, output schema. Totally different than what I’m going on about.

Re: JC – JSONifies the output of many CLI tools

#120

Post by speed_spread »

naikrovek wrote: post #108
runlevel1 wrote: post #90

Earlier quoted context omitted.

That reminds me of something I've wanted for quite a while: A ringbuffer filetype. Similar to a named pipe file (see: fifo(7)[^1]), but without consuming the contents on read and automatically rotating out the oldest lines. Of course, there would be some complexities around handling read position as lines are being rotated out from under you. [1]: https://linux.die.net/man/7/fifo

> Of course, there would be some complexities around handling read position as lines are being rotated out from under you. that seems solvable to me, but multiple simultaneous readers... that seems like it might be a bit more challenging...

Only allow sequential reads. Each reader gets it's own cursor. If writer catches up to slowest reader (buffer full) it bumps it up before it writes. Clients can query current cursor position to know if they've been bumped. Or something like that.
Post reply on HN