Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For the Love of Pipes

blog.jessfraz.com

Post reply on HN
Print view View original

291–300 of 323 posts

Re: For the Love of Pipes

#291

Post by anthk »

wglb wrote: post #180

This is cool and useful, but not all unix programs follow this convention: * find * cal * vi * emacs * ls These don't use one of standard input/standard output. (edited) and are not fully pipeable. I don't recall seeing a list of programs--tools, in the original description--that distinguish between pipeable and not-pipeable programs. Also, none of the corrective cat/grep code in these threads point out that grep in…

- Find has some switches to sanityze the output

- Cal can be parsed (it's shows in the Unix Programming Environment, from 1983)

- Vi is a visual editor, it can be used as a front-end for ed/ex commands for I/O anyway. Kinda like the acme(1) of its day. You have both :w and :r. And, hint: it can input text from external pipes.

- Emacs is not Unix

- ls(1) is not meant to be parsed on files' content, that's the shell globbing for.

Re: For the Love of Pipes

#292

Post by anthk »

pdkl95 wrote: post #83
mothsonasloth wrote: post #51

[Quote] The Unix philosophy is documented by Doug McIlroy as: Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”. Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.…

> The Unix philosophy is documented by Doug McIlroy as TaoUP has a longer discussion[1] of the Unix philosophy, which includes Rob Pike's and Ken Thompson's comments on the philosophy. [1] http://www.catb.org/esr/writings/taoup/html/ch01s06.html "Those who don't understand Unix are condemned to reinvent it, poorly." (Henry Spencer)

Avoid TAOUP, is really bad. Most of the lore have stolen from places as LISP and VAX communities. ESR is to Pike and Ken as alien as X11 itself.

Re: For the Love of Pipes

#293

Post by anthk »

arendtio wrote: post #123
yaakushi wrote: post #85

I'm probably nitpicking, but if you're using cat to pipe a single file into the sdtin of another program, you most likely don't need the cat in the first place, you can just redirect the file to the process' stdin. Unless, of course, you're actually concatenating multiple files or maybe a file and stdin together. Disclaimer: I do cat-piping myself quite a bit out of habit, so I'm not trying to look down at the author…

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

awk can do all of that except sed. And I am not sure about the last. No need to wc ($NF in AWK, if I can recall), no need for grep, you have the /match/ statement, with regex too.

Re: For the Love of Pipes

#294

Post by anthk »

ncallaway wrote: post #138
Hello71 wrote: post #134

Earlier quoted context omitted.

This is a very silly way of writing it though. grep|sed can almost always be replaced with a simple awk: awk '/^x/ { sub("a", "b"); print $2; }' foo.txt. This way, the whole command fits on one line. If it doesn't, put your awk script in a separate file and simply call it with "awk -f myawkscript foo.txt".

I would disagree that their way of writing it is silly. It is instantly plainly obvious to me what each step of their shell script is doing. While I can absolutely understand what your shell script does after parsing it, it's meaning doesn't leap out at me in the same way. I would describe the prior shell script as more quickly readable than the one that you've listed. So, perhaps it's not a question of one being mor…

That's bullshit. On million files, spawning subproceses could add a several lag per pipe. This way, awk runs straightly. AWK is base knowledge on Unix. Shell scripts, specially the bloated bash, not. Readable? Priorities? I've done quicker jobs with awk and windows binary logs with strings(1) with the more than, literally, 100+ lines from newbies.

If people learn more about its tools instead of adding bloat unnedeedly because of ignorance OLD and ULTRADOCUMMENTED UNIX tools, - -cough, GNU folks and GNU/Linux distros- most of the crappified scripts and (fake) "usability" switches woudn't even exist.

Such as column, tr, "objdum -x $FILE | grep NEEDED" instead of ldd, tput for nice dialog like interfaces in shell, join, fmt, and so on.

Also, makefiles/m4. They cut down times exponentially.

Re: For the Love of Pipes

#295

Post by __MatrixMan__ »

cryptonector wrote: post #266
__MatrixMan__ wrote: post #149

Earlier quoted context omitted.

I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work. Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a…

This is interesting, yes. If the shell could infer the content type of data demanded or output by each command in a pipeline, then it could automatically insert type coercion commands or alter the options of commands to produce the desired content types. You're right that it is in fact possible for a command to find the preceding and following commands using /proc, and figure out what content types they produce / wan…

My ultimate use case is a contrived environment where I have the luxury of ignoring otherwise blatant feature-gaps--such as compatibility with other tools (like curl). I've come to the same conclusions about why that might be tricky, so I'm calling it a version-two problem.

I notice that function composition notation; that is, the latter half of:

> f(g(x)) = (f o g)(x)

resembles bash pipeline syntax to a certain degree. The 'o' symbol can be taken to mean "following". If we introduce new notation where '|' means "followed by" then we can flip the whole thing around and get:

> f(g(x)) = (f o g)(x) = echo 'x' | g | f

I want to write some set of mathematically interesting functions so that they're incredibly friendly (like, they'll find and fix type mismatch errors where possible, and fail in very friendly ways when not). And then use the resulting environment to teach a course that would be a simultaneous intro into both category theory and UNIX.

All that to say--I agree about finding the magic a little distasteful, but if I play my cards right my students will only realize there was magic in play after they've taken the bait. At first it will all seem so easy...

Re: For the Love of Pipes

#296

Post by cryptonector »

__MatrixMan__ wrote: post #295
cryptonector wrote: post #266

Earlier quoted context omitted.

This is interesting, yes. If the shell could infer the content type of data demanded or output by each command in a pipeline, then it could automatically insert type coercion commands or alter the options of commands to produce the desired content types. You're right that it is in fact possible for a command to find the preceding and following commands using /proc, and figure out what content types they produce / wan…

My ultimate use case is a contrived environment where I have the luxury of ignoring otherwise blatant feature-gaps--such as compatibility with other tools (like curl). I've come to the same conclusions about why that might be tricky, so I'm calling it a version-two problem. I notice that function composition notation; that is, the latter half of: > f(g(x)) = (f o g)(x) resembles bash pipeline syntax to a certain degr…

The magic /proc thing is a very interesting challenge. Trust me, since I read your comments I've thought about how to implement, though again, it's not the sort of thing I'd build for a production system, just a toy -- a damned interesting one. And as a tool for teaching how to find your way around an OS and get the information you need, it's very nice. There's three parts to this: a) finding who's before and after the adapter in the pipe, b) figuring out how to use that information to derive content types, c) match impedances. (b) feels mundane: you'll have a table-driven approach to that. Maybe you'll "taste" the data when you don't find a match in the table? (c) is not always obvious -- often the data is not structured. You might resort to using extended file attributes to store file content-type metadata (I've done this), and maybe you can find the stdin or other open files of the left-most command in a pipeline, then you might be able to guesstimate the content type in more cases. But obviously, a sed, awk, or cut, is going to ruin everything. Even something like jq will: you can't assume the output and input will be JSON.

At some point you just want a Haskell shell (there is one). Or a jq shell (there is something like it too).

As to the pipe symbol as function composition: yes, that's quite right.

Re: For the Love of Pipes

#297

Post by wahern »

anthk wrote: post #291
wglb wrote: post #180

This is cool and useful, but not all unix programs follow this convention: * find * cal * vi * emacs * ls These don't use one of standard input/standard output. (edited) and are not fully pipeable. I don't recall seeing a list of programs--tools, in the original description--that distinguish between pipeable and not-pipeable programs. Also, none of the corrective cat/grep code in these threads point out that grep in…

- Find has some switches to sanityze the output - Cal can be parsed (it's shows in the Unix Programming Environment, from 1983) - Vi is a visual editor, it can be used as a front-end for ed/ex commands for I/O anyway. Kinda like the acme(1) of its day. You have both :w and :r. And, hint: it can input text from external pipes. - Emacs is not Unix - ls(1) is not meant to be parsed on files' content, that's the shell gl…

FWIW, POSIX specifies the ls output format for -l and several other flags. For example,

  If the -l option is specified, the following information
  shall be written for files other than character special and
  block special files:

  "%s %u %s %s %u %s %s\n", , ,
    , , , ,
There's no stat command in POSIX. More practically, the BSD and GNU versions of stat are completely incompatible. If you want to query a file's size or other metadata from a portable shell script, you need to parse the output of ls.

Re: For the Love of Pipes

#298

Post by __MatrixMan__ »

laumars wrote: post #276
__MatrixMan__ wrote: post #217

Earlier quoted context omitted.

That sounds reasonable, I'll look into it--thanks. I was imagining an algorithm where each pipeline-aware utility can derive port numbers to use to talk/listen to its neighbors. I may be able to use http content negotiation wholesale in that context.

I've been trying to solve the exact same problem with my shell too. It's pipes are typed and all the builtin commands can than automatically decode those data types via shared libraries. So commands don't need to worry about how to decode and re-encode the data. This means that JSON, YAML, TOML, CSV, Apache log files, S-Expressions and even tabulated data from `ps` (for example) can all be transparently handled the s…

> ...with my shell too...

I was hoping to stick with bash or zsh, and just write processes that somehow communicate out of band, but I think we're still up against the same problem.

One idea I had was that there's a service running elsewhere which maintains this directed graph (nodes = types, edges = programs which take the type of their "from" node and return the type of their "two" node). When a pipeline is executed, each stage pauses until type matches are confirmed--and if there is a mismatch then some path finding algorithm is used to find the missing hops.

So the user can leave out otherwise necessary steps, and as long as there is only one path through the type graph which connects them, then the missing step can be "inserted". In the case of multiple paths, the error message can be quite friendly.

This means keeping your context small enough, and your types diverse enough, that the type graph isn't too heavily connected. (Maybe you'd have to swap out contexts to keep the noise down.) But if you have a layer that's modifying things before execution anyway, then perhaps you can have it notice the ssh call and modify it to set up a listener. Something like:

User Types:

    local-command | ssh user@host "remote-command"
Shell runs:

    local-command | ssh user@host "pull_metadata_from -r  | remote-command"
Where pull_metadata_from phones home to get the metadata, then passes along the data stream untouched.

Also, If you're writing the shell anyway then you can have the pipeline run each process in a subshell where vars like TYPE_REGISTRY_IP and METADATA_INBOUND_PORT are defined. If they're using the network to type-negotiate locally, then why not also use the network to type-negotiate through an ssh tunnel?

This idea is, of course, over-engineered as hell. But then again this whole pursuit is.

Re: For the Love of Pipes

#299

Post by bigger_cheese »

timvisee wrote: post #61

Cool, the pipe command must be one of the most essential things in Unix/Linux based systems. I would have loved to see some awesome pipe examples though.

Back when I first started using Linux you could pipe random data to /dev/dsp and the speakers would emit various beeps. Used to be a pretty cool trick I think when ASLA came out it stopped working.

Re: For the Love of Pipes

#300

Post by yesenadam »

geofft wrote: post #136
Hello71 wrote: post #134

Earlier quoted context omitted.

This is a very silly way of writing it though. grep|sed can almost always be replaced with a simple awk: awk '/^x/ { sub("a", "b"); print $2; }' foo.txt. This way, the whole command fits on one line. If it doesn't, put your awk script in a separate file and simply call it with "awk -f myawkscript foo.txt".

I use awk in exactly this way personally, but, awk is not as commonly readable as grep and sed (in fact, that use of grep and sed should be pretty comprehensible to someone who just knows regular expressions from some programming languages and very briefly glances at the manpages, whereas it would be difficult to learn what that awk syntax means just from e.g. the GNU awk manpage). So, just as you could write a Perl…

Not sure why you say grep and sed are more readable than awk! (not sure what 'commonly readable' means). Or that even that particular line in awk is harder to understand than the grep and sed man pages. The awk manpage even has examples, including print $2. The sed manpages must be the most impenetrable manpages known to 'man', if you don't already understand sed. (People might already know s///g because 99% of the time, that's all sed is used for.)
Post reply on HN