Live data from Hacker News

Bringing the Unix philosophy to the 21st century (2019)

blog.kellybrazil.com

31–40 of 151 posts

Re: Bringing the Unix philosophy to the 21st century (2019)

#31
post #21

Earlier quoted context omitted.

I don’t think the issue is that it’s hard to manually parse. The problem is that it’s hard for someone else to read your ad-hoc parser years later and reason about what you did if they need to modify it. Disclaimer: I am the author of the article and JC.

This is even more true for the ungodly long `jq` incantations that people write. It's like I get it, the old way is ugly and not always easy to decipher but at least it's shorter and your chances of understanding it are better. I've had both -- the classic piped chain of UNIX commands and various JSON-producing tools piped to `jq`. The former were still easier to work with.

Yes, I have seen those too! That’s why I also wrote Jello, which is like jq but uses pure python without the boilerplate. Python is nearly universal now and typically easy to read, though more verbose. Jq is just as much a write-once tool as awk and perl for more complex queries. For simple attribute calls, though, it’s both terse and readable.

Re: Bringing the Unix philosophy to the 21st century (2019)

#32
post #19

So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…

[deleted]

Re: Bringing the Unix philosophy to the 21st century (2019)

#33

Uh ... what about when I don't want to load the entire stream into memory before the next stage starts running? Are these implicit json arrays that stream out? Or do we now have incompatible json shell tools and streaming text tools as a permanent fixture? I'm excited about structured output ideas, but json? I'd much rather have streams of whitespace separated words than json. That's in that "No type system is better…

For the first point, there's no reason why we can't use JSON array wrapped output. In fact, we likely should use something like this for uniformity. Loading everything into memory is also not too problematic assuming a working swap space and reasonably well-architectured output schema.

For the second point, whitespace sensitivity is the one mistake that greatly pisses me off with Unix. I should be able to pass arguments and filenames with as many spaces as I want. We are in the 21st century and occasionally do use spaces in filenames.

Re: Bringing the Unix philosophy to the 21st century (2019)

#34
post #23

Why not go all the way and use a format capable of expressing code and data? I refer, of course, to S-expressions. They also have the benefit of properly handling numbers. Some might look at the absence of maps as a negative, but I think alists are preferable anyway due to their constant ordering.

I'd rather avoid mixing the two. Shell injection is already a danger.

Re: Bringing the Unix philosophy to the 21st century (2019)

#35

Uh ... what about when I don't want to load the entire stream into memory before the next stage starts running? Are these implicit json arrays that stream out? Or do we now have incompatible json shell tools and streaming text tools as a permanent fixture? I'm excited about structured output ideas, but json? I'd much rather have streams of whitespace separated words than json. That's in that "No type system is better…

I agree JSON is probably not right for every type of program output, but the age of web APIs has shown us that is probably great or adequate 90% of the time. If something is spewing out long lines of data I think JSON Lines would be a good option so you don’t need to read the whole structure into RAM. But any other structured output that has a healthy community and ecosystem supporting it would be better than just space delimited lines, or worse - groups of lines you need to deal with.

Re: Bringing the Unix philosophy to the 21st century (2019)

#36

Uh ... what about when I don't want to load the entire stream into memory before the next stage starts running? Are these implicit json arrays that stream out? Or do we now have incompatible json shell tools and streaming text tools as a permanent fixture? I'm excited about structured output ideas, but json? I'd much rather have streams of whitespace separated words than json. That's in that "No type system is better…

I agree with you in broad strokes, but as a piece of anecdata: I've had a lot of success building tools that emit and consume JSONL[1] instead of entire JSON documents. JSONL preserves the Unix pipeline's inherently parallel design (people tend to forget this, even when waxing about the Unix philosophy!) but gives us all of the nice typing of a JSON stream.

That being said, I too will take a `sed` or `awk` one-liner over some of the `jq` monstrosities that I've seen.

[1]: https://jsonlines.org/

Re: Bringing the Unix philosophy to the 21st century (2019)

#37

I was literally just thinking about this a few days ago. I'm super excited by https://www.nushell.sh/ . I think they are hitting on an order of magnitude improvement paradigm of shells that fit very nicely with the theme of this article.

https://www.nushell.sh/book/ This goes against the philosophy explicitly mentioned in OP's article. E.g. avoid tabular formats. This is systemd against init again. A powerful, but overreaching shell that becomes unreplaceable and bloated with concerns. In constrast, traditional *nix/GNU programs work well, and interact well, in every shell.

You see traditional Unix/GNU programs as working well and interacting will. I see a programming paradigm designed for six-char identifiers, not designed for whitespaces and punctuation in names, running in something that emulates a physical terminal that emulated a paper-based terminal. Terseness is there, but ergonomics of use for non-utter-experts could be improved.

I think that the shell paradigm could be so much better. If there's a momentum to make a change from that, I'll jump on the bandwagon. Any change is better than the current state of the shell.

Re: Bringing the Unix philosophy to the 21st century (2019)

#38
Related, I am wondering if somebody is aware of a project to wrap cli utilies in a minimal 'TUI', along the lines of this: https://github.com/chriskiehl/Gooey , but staying in the terminal. You'd pass the name of the program to run as the first argument. Haven't thought this through fully.

EDIT: well, some discussion about this: https://github.com/chriskiehl/Gooey/issues/296

Re: Bringing the Unix philosophy to the 21st century (2019)

#39
post #18

That `jc` command that converts non-json output to json is neat. But, um, seems slightly kludgy to be obvious. What if the `ifconfig` format changes slightly. I suppose its a stop-gap until all commands have a json output option.

True. When I try it with `date` on Debian Sid I get parser error:

    $ jc -p date
    jc:  Error - date parser could not parse the input data. Did you use the correct parser?
             For details use the -d or -dd option.

Re: Bringing the Unix philosophy to the 21st century (2019)

#40
post #39
post #18

That `jc` command that converts non-json output to json is neat. But, um, seems slightly kludgy to be obvious. What if the `ifconfig` format changes slightly. I suppose its a stop-gap until all commands have a json output option.

True. When I try it with `date` on Debian Sid I get parser error: $ jc -p date jc: Error - date parser could not parse the input data. Did you use the correct parser? For details use the -d or -dd option.

Odd. What locale are you using? Should work fine with C or en_UTF8, per the readme.
Post reply on HN