Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For the Love of Pipes

blog.jessfraz.com

Post reply on HN
Print view View original

271–280 of 323 posts

Re: For the Love of Pipes

#271

Post by laumars »

msla wrote: post #257
laumars wrote: post #240

Earlier quoted context omitted.

I don’t like that at all. That creates a subshell and is also less readable than input | grep foo | sed ...

That specific example is less readable, but I do like being able to do this: diff and get a sensible result. And sometimes programs just refuse to read from stdin but do just fine with an unseekable file on the command line. True, you do have this: input | recalcitrant_program /dev/stdin ... but it's a bit of a tossup as to which one's more readable at this point. They're both relying on advanced shell functionality.

> That specific example is less readable, but I do like being able to do this:

> diff > and get a sensible result.

That is called process substitution and is exactly the kind of use case that it's designed for. So yes, process substitution does make sense there.

> input | recalcitrant_program /dev/stdin

> ... but it's a bit of a tossup as to which one's more readable at this point. They're both relying on advanced shell functionality.

There's no tossup at all. Process substitution is easily more readable than your second example because you're honouring the normal syntax of that particular command's parameters rather than kludging around it's lack of STDIN support.

Also I wouldn't say either example is using advanced shell functionalities either. Process substitution (your first example) is a pretty easy thing to learn and your second example is just using regular anonymous pipes (/dev/stdin isn't a shell function, it's a proper pseudo-device like /dev/random and /dev/null) thus the only thing the shell is doing is the same pipe described in this threads article (with UNIX / Linux then doing the clever stuff outside of the shell).

Re: For the Love of Pipes

#272

Post by dan-robertson »

iamwil wrote: post #265
nojvek wrote: post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

What happens when there are upstream changes to the objects? Does everything downstream just need to change, and hence, the upstream objects returned by programs need to be changed with care? Or is it using something like protobuf, where fields are only additive, but never deleted, for backwards compatibility? Or are the resulting chain of pipes so short lived, it doesn't matter?

What happens when you rely on non standard behaviour of unix tools, or just non-posix tools that change from beneath you?

I’m not saying that this makes powershell pipes better/worse, just that this problem isn’t unique. Microsoft tends to be reasonably committed to backwards compatibility but I don’t know the answer to the question

Re: For the Love of Pipes

#273

Post by dan-robertson »

yaakushi wrote: post #85

I'm probably nitpicking, but if you're using cat to pipe a single file into the sdtin of another program, you most likely don't need the cat in the first place, you can just redirect the file to the process' stdin. Unless, of course, you're actually concatenating multiple files or maybe a file and stdin together. Disclaimer: I do cat-piping myself quite a bit out of habit, so I'm not trying to look down at the author…

For interactive use, I would like to point out that even better than this use of cat is less. If you pipe less into something then it forgets it’s interactive behaviour and works like cat on a single file. So:

  $ less foo | bar
Is similar too:

  $ bar
Except that less is typically more clever than that and might be more like:

  $ zcat foo | bar
Depending on the file type of foo.

Re: For the Love of Pipes

#274

Post by paavoova »

nojvek wrote: post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

The downside of objects is performance due to the e.g. increased overhead a file object carries. Plus the exact same can be argued with everything being bytes/text on Unix, in that it make everything simple but versatile.

> get some field out of `ls`

If you're parsing the output of ls, you're doing it wrong. Filenames can contain characters such as newlines, etc, and so you generally want to use globbing/other shell builtins, or utils like find with null as the separator instead of newline.

Re: For the Love of Pipes

#276

Post by laumars »

__MatrixMan__ wrote: post #217
jshen wrote: post #163

Earlier quoted context omitted.

I wonder if something like HTTP’s content negotiation is a good model for this.

That sounds reasonable, I'll look into it--thanks. I was imagining an algorithm where each pipeline-aware utility can derive port numbers to use to talk/listen to its neighbors. I may be able to use http content negotiation wholesale in that context.

I've been trying to solve the exact same problem with my shell too. It's pipes are typed and all the builtin commands can than automatically decode those data types via shared libraries. So commands don't need to worry about how to decode and re-encode the data. This means that JSON, YAML, TOML, CSV, Apache log files, S-Expressions and even tabulated data from `ps` (for example) can all be transparently handled the same way and converted from one to another without the tools ever needing to know how to marshal nor unmarshal that data. For example: you could take a JSON array that's not been formatted with cartridge returns and still grep through it item by item as if it was a multi-line string.

However the problem I face is how do you pass that data type information over a pipeline from tools that exist outside of my shell? It's all well and good having builtins that all follow that convention but what if someone else wants to write a tool?

My first thought was to use network sockets, but then you break piping over SSH, eg:

    local-command | ssh user@host "| remote-command"
My next thought was maybe this data should be in-lined - a bit like how ANSI escape sequences are in-lined and the terminals don't render them as printable characters. Maybe something like the following as a prefix to STDIN?

    $SHELL
But then you have the problem of tainting your data if any tools are sent that prefix in error.

I also wondered if setting environmental variables might work but that also wouldn't be reliable for SSH connections.

So as you can see, I'm yet to think up a robust way of achieving this goal. However in the case of builtin tools and shell scripts, I've got it working for the most part. A few bugs here and there but it's not a small project I've taken on.

If you fancy comparing notes on this further, I'm happy to oblige. I'm still hopeful we can find a suitable workaround to the problems described above.

Re: For the Love of Pipes

#277

Post by mirimir »

GuB-42 wrote: post #161
arendtio wrote: post #123

Earlier quoted context omitted.

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

IMHO, shell scripts are a minefield and if you want something readable and portable, this is also the wrong technology. They are convenient though. They are like the Excel macros of the UNIX world. Now back to the topic of "cat", which is a great example of why shell scripts are minefields. Replace "foo.txt" with a user supplied variable, let's call it "$F". It becomes cat $F | blah_blah... I mean cat "$F" | blah_bla…

> Replace "foo.txt" with a user supplied variable, let's call it "$F". It becomes cat $F | blah_blah... I mean cat "$F" | blah_blah, first trap, but everyone knows that.

I think that you forgot to edit the "I mean" to "echo $F" :)

Re: For the Love of Pipes

#278

Post by Pawamoy »

dan-robertson wrote: post #272
iamwil wrote: post #265

Earlier quoted context omitted.

What happens when there are upstream changes to the objects? Does everything downstream just need to change, and hence, the upstream objects returned by programs need to be changed with care? Or is it using something like protobuf, where fields are only additive, but never deleted, for backwards compatibility? Or are the resulting chain of pipes so short lived, it doesn't matter?

What happens when you rely on non standard behaviour of unix tools, or just non-posix tools that change from beneath you? I’m not saying that this makes powershell pipes better/worse, just that this problem isn’t unique. Microsoft tends to be reasonably committed to backwards compatibility but I don’t know the answer to the question

I think they were talking about objects at execution time, not version compatibility.

Re: For the Love of Pipes

#279

Post by pka »

barrkel wrote: post #241
chriswarbo wrote: post #169

Earlier quoted context omitted.

> list.select { |x| x.foo > 10 }.map { |x| x.bar }... Forgive me if I'm misreading this syntax, but to me this looks like plain old function composition: a call to `select` (I assume that's like Haskell's `filter`?) composed with a call to `map`. No monad in sight. As I mentioned, monads are more about collapsing structure. In the case of lists this could be done with `concat` (which is the list implementation of mon…

Nope, it's not. It's Ruby, and the list could be an eager iterator, an actual list, a lazy iterator, a Mabye (though it would be clumsy in Ruby), etc. And monads are not "more about collapsing structure". They are just a design pattern that follows a handful of laws. It seems like you're mistaking their usefulness in Haskell for what they are. A lot of other languages have monads either baked in or an element of the…

The things you listed may form a monad, but your list/iterator transformation doesn't use monadic composition.

> list.select { |x| x.foo > 10 }.map { |x| x.bar }

Where's `bind` or `join` in that example?

Re: For the Love of Pipes

#280

Post by anthk »

nojvek wrote: post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

Never parse ls.
Post reply on HN