Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bringing the Unix philosophy to the 21st century (2019)

blog.kellybrazil.com

Post reply on HN
Print view View original

101–110 of 151 posts

Re: Bringing the Unix philosophy to the 21st century (2019)

#101

Post by kbrazil »

guenthert wrote: post #97
zozbot234 wrote: post #12

Earlier quoted context omitted.

> More generally, text is great for loose coupling; structured objects, less so. I disagree. Parsing "loose coupled" text and converting it to the format that a different tool expects is a rather non-trivial problem, and one that's often poorly specified to begin with; converting structured outputs is generally straightforward in comparison.

Sure, converting (or interpreting) structured data is easier then unstructured. I doubt anyone is arguing that. I interpret op's statement however in that unstructured data is easier to couple unrelated tools. Tools perhaps which haven't written yet, by teams not knowing of each other and hence are unable to agree on a structure.

I feel like if that were true then you’d have the majority of REST APIs delivering unstructured output instead of structured JSON, XML, CSV, etc. The market has spoken - there is true utility in structured text output.

Re: Bringing the Unix philosophy to the 21st century (2019)

#102

Post by yyyk »

The idea is good, but the approach cannot execute sufficiently. First, jc has to individually support each program and program mode, so improvements are bottlenecked on jc. Everytime you load a program you need to load all jc parsers. I also wonder how resistant the parsing is to surprises (filenames with bracket characters in ls output for example).

Second, this does not handle interprogram piping (e.g. 'find . | xargs dosomething'). We have filtering to console but little piping (we can map the json output to a single column and pipe that, using the textual interface with all its deficiencies).

To be clear, the author has no choice (rewriting all userland is a daunting concept), but a proper approach for 21th century Unix would be for all utilities to output and receive json (or whatever format is chosen) natively, and that the console would then be able to mange that into a textual format for the user.

Re: Bringing the Unix philosophy to the 21st century (2019)

#103

Post by xelxebar »

athrowaway3z wrote: post #47
c0l0 wrote: post #19

So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…

I know enough awk to know it solves my problem, but use it to little to remember how, and the development experience is never very clean. What i would like is an cli awk ide. (Rolls of the tongue right? ) Ideally i could write ... | awk-ide "scripts/thing.awk" | ... If it exists, run it. Otherwise have it open an editor session that runs an awk program as i type it on some buffered data, shows the result, and have so…

What about something like

  while inotifywait --event modify,move_self,delete_self script.awk; do awk -f script.awk input.txt; done
in a tmux pane on the right with vim/emacs/whatever on the left?

Note: You need to watch the move_self and delete_self events, since editors might use some file move/deletion dance instead of a direct modify to help prevent data loss in the event of a crash.

Re: Bringing the Unix philosophy to the 21st century (2019)

#104

Post by kbrazil »

athrowaway3z wrote: post #47
c0l0 wrote: post #19

So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…

I know enough awk to know it solves my problem, but use it to little to remember how, and the development experience is never very clean. What i would like is an cli awk ide. (Rolls of the tongue right? ) Ideally i could write ... | awk-ide "scripts/thing.awk" | ... If it exists, run it. Otherwise have it open an editor session that runs an awk program as i type it on some buffered data, shows the result, and have so…

I wrote something similar to this to query JSON and JSON lines with python instead of awk for text. It’s called Jellex (Jello Explorer) which is a TUI front-end to Jello. Jello is a python analog to JQ.

https://github.com/kellyjonbrazil/jellex

Re: Bringing the Unix philosophy to the 21st century (2019)

#105

Post by wrycoder »

chillfox wrote: post #100
yesenadam wrote: post #87

Earlier quoted context omitted.

The original 1988 book The AWK Programming Language by A, W and K is, in my opinion, one of the finest pieces of documentation ever written, on any subject. A joy to read and be instructed by.

And $722 AUD on Amazon. Did they never make a second printing?

https://archive.org/details/pdfy-MgN0H1joIoDVoIC7

Re: Bringing the Unix philosophy to the 21st century (2019)

#106

Post by lambdaba »

laumars wrote: post #86
laumars wrote: post #64

Earlier quoted context omitted.

There are better shells out there for handling structural data, like Murex and Elvish

No seriously, there really are. I appreciate it requires learning a new tool but rather than downvoting me how about those unconvinced ask me questions instead? I’m happy to answer. While there will always be a need to keep Bash around for comparability, there are a plethora of other tools out there that solve many of the shortcomings of POSIX.

Are you using one of these exclusively? Which one have you chosen and why?

Re: Bringing the Unix philosophy to the 21st century (2019)

#107

Post by kumarsw »

With all the discussion here about object shells, it's a bit surprising that the project page doesn't make stronger emphasis that this (jc) is also a Python library that can take in the output of subprocess.Popen rather than fooling about with the mess that is jq. With that said, Python isn't especially nice for calling external programs. I can't wait for nushell/oil shell/elvish to hit maturity soon enough.

Re: Bringing the Unix philosophy to the 21st century (2019)

#110

Post by sennight »

kbrazil wrote: post #75
phendrenad2 wrote: post #70

"Up until about 2013 it made just as much sense as anything to assume unstructured text was a good way to output data at the command line..." But in 2013 a certain data format called JSON was standardized as ECMA-404..." "Had JSON been around when I was born in the 1970’s Ken Thompson and Dennis Ritchie may very well have embraced it as a recommended output format to help programs “do one thing well” in a pipeline."…

I don’t follow. Your examples are of data formats standardized in the 80’s while Unix was developed in the 60’s and 70’s. JSON even existed before 2013, but the fact that it became a standard in addition to being popular is the point I was making.

I've spent a lot of time reverse engineering long dead hardware (and forgotten protocols), which means I've also spent a lot of time reading ancient papers generated by IBM, Bell, Arpa (so many long gone DoD programs), etc. So I sometimes have a hard time distinguishing between what is common knowledge and what is something that only myself and maybe a handful of others care about...

That said, you know that every potential variation of every possible approach to accomplish what I presume your objectives are has already been exhaustively explored, documented, implemented, and finally abandoned by organizations with functionally limitless resources (aka open ended government contracts) - 50 years ago? Have you considered leveraging some of that work? I don't think many people know about the massive amount of work already done - that was just abandoned for a variety of reasons: reasons that often no longer apply, and very rarely have anything to do with the technology's utility. For example: everybody here knows about the OSI model and how sparsely filled out it is - but did you know that there is a layer set aside to do exactly what you are talking about, and that it just isn't being used? Thats right, #6, the presentation layer - specifically the virtual terminal protocol: which is where the designers wanted the object exchange and structured data to go... not in a mess of json one layer up. The VTP was outlined in several papers going back to at least '72. You could also lean on the Airforce's work for your data model, they ran that to ground pretty thoroughly with IDEF. You've also got a huge amount of free work from IBM when it comes to structured documents, and architecture that lends itself to semantic reasoning.

Anyway, my larger point is that instead of exasperating the issue of wastefully bloated software teetering on increasingly high layers of abstractions (do a stack trace and consider the insanity of it), maybe the way to really improve our circumstances has already been discovered and then lost for a time. It would be no more difficult than trying to make kornshell-json a thing.

A Survey of Terminal Protocols (1979) DOI 10.1016/0376-5075(79)90001-1

Computer Network Architectures and Protocols (1983) DOI 10.1007/978-1-4615-6698-4

IDEF: https://en.wikipedia.org/wiki/IDEF

IBM's various journals (pre-'00) are also worth reading.

Post reply on HN