Live data from Hacker News

For the Love of Pipes

blog.jessfraz.com

161–170 of 323 posts

Re: For the Love of Pipes

#161

I'm probably nitpicking, but if you're using cat to pipe a single file into the sdtin of another program, you most likely don't need the cat in the first place, you can just redirect the file to the process' stdin. Unless, of course, you're actually concatenating multiple files or maybe a file and stdin together. Disclaimer: I do cat-piping myself quite a bit out of habit, so I'm not trying to look down at the author…

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

IMHO, shell scripts are a minefield and if you want something readable and portable, this is also the wrong technology. They are convenient though. They are like the Excel macros of the UNIX world.

Now back to the topic of "cat", which is a great example of why shell scripts are minefields.

Replace "foo.txt" with a user supplied variable, let's call it "$F". It becomes cat $F | blah_blah... I mean cat "$F" | blah_blah, first trap, but everyone knows that.

Now, if F='-n', second trap. What you think is a file will be considered an option and cat will wait for user input, like when no file is given. Ok, so you need to do cat -- "$F" | blah_blah.

That should be OK in every case now, but remember that "cat" is just another executable, or maybe a builtin. For some reason, on your system "cat --" may not work, or some asshat may have added "." in your PATH and you may be in a directory with a file named "cat". Or maybe some alias that decides to add color.

There are other things to consider, like your locale that may mess up you output with comas instead of decimal points and unicode characters. For that reason, you need to be very careful every time you call a command and even more so if you pipe the output.

For that reason, I avoid using "cat" in scripts. It is an extra command call and all the associated headaches I can do without.

Re: For the Love of Pipes

#162

has there been proposals/research on having non linear / branching pipe syntax ? a |1 b | c | d, |1 e aka piping a to both b,c,d and e branches ?

https://en.m.wikipedia.org/wiki/Named_pipe

that's a building block, not a good ~interface. Kinda like redirections before pipe were added to shells

Re: For the Love of Pipes

#163

Earlier quoted context omitted.

To be fair, the same critisim could be used for a socket? I think the issue is that some people want pipes to be something magical that connects their software, not a dumb connection between them.

I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work. Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a…

I wonder if something like HTTP’s content negotiation is a good model for this.

Re: For the Love of Pipes

#165

has there been proposals/research on having non linear / branching pipe syntax ? a |1 b | c | d, |1 e aka piping a to both b,c,d and e branches ?

Smalltalk's "method cascading" (the `;` operator) did roughly that. I don't think you're forking an entire sequence of methods though, it just allows performing operations on the same root object e.g. a foo ; bar ; baz would sequentially send "foo", "bar" then "baz" to "a", then would return the result of the last call. The ability to fork iterators (which may be the semantics you're actually interested in) also exis…

I see.. but I wonder if st syntax allows for more than methods:

    a (foo duh meh) 

Re: For the Love of Pipes

#167

has there been proposals/research on having non linear / branching pipe syntax ? a |1 b | c | d, |1 e aka piping a to both b,c,d and e branches ?

I don't think the commandline is well suited to constructing or reading anything more complicated than a linear pipe. What you'd want for that kind of workflow is a 2d layout.

That's actually the question underneath.. can we write simple graphs on a command line :)

Re: For the Love of Pipes

#168
post #154

Earlier quoted context omitted.

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

I find something like this: grep '^x' to be very readable, as the flow is still visually apparent based on punctuation.

I don't like this style at all. If you're following the pipeline, it starts in the middle with "input", goes to the left for the grep, then to the right (skipping over the middle part) to sed.

     cat input | grep '^x' | sed 's/foo/bar/g'
Is far more readable, in my opinion. In addition, it makes it trivial to change the input from a file to any kind of process.

I'm STRONGLY in favor of using "cat" for input. That "useless use of cat" article is pretty dumb, IMHO.

Re: For the Love of Pipes

#169
post #80

Earlier quoted context omitted.

I'm not sure I agree with this. Function composition is more directly comparable to pipes, whereas I tend to think of monads as collapsing structure (i.e. `join :: m (m a) -> m a`)

Some of the most common monads (especially lists and other collection-like things) feel very much like pipes. list.select { |x| x.foo > 10 }.map { |x| x.bar }... Etc. I wouldn't make the same argument about the IO monad, which I think more in terms of a functional program which evaluates to an imperative program. But most monads are not like the IO monad, in my experience at least.

> list.select { |x| x.foo > 10 }.map { |x| x.bar }...

Forgive me if I'm misreading this syntax, but to me this looks like plain old function composition: a call to `select` (I assume that's like Haskell's `filter`?) composed with a call to `map`. No monad in sight.

As I mentioned, monads are more about collapsing structure. In the case of lists this could be done with `concat` (which is the list implementation of monad's `join`) or `concatMap` (which is the list implementation of monad's `bind` AKA `>>=`).

Re: For the Love of Pipes

#170

[Quote] The Unix philosophy is documented by Doug McIlroy as: Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”. Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.…

From 1978, and still applicable to microservices today.
Post reply on HN