Live data from Hacker News

For the Love of Pipes

blog.jessfraz.com

231–240 of 323 posts

Re: For the Love of Pipes

#231
post #26

Earlier quoted context omitted.

Monads are effectively pipes; the monad controls how data flows through the functions you put into the monad, but the functions individually are like individual programs in a pipe.

I'm not sure I agree with this. Function composition is more directly comparable to pipes, whereas I tend to think of monads as collapsing structure (i.e. `join :: m (m a) -> m a`)

tiny clarification: Function composition is just function composition. Pipes are a different syntax for function application.

see https://news.ycombinator.com/item?id=18971451

Re: For the Love of Pipes

#232
post #224
post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

She is actually employed at GitHub now. https://twitter.com/jessfraz

Which is owned by...

Re: For the Love of Pipes

#233

I'm probably nitpicking, but if you're using cat to pipe a single file into the sdtin of another program, you most likely don't need the cat in the first place, you can just redirect the file to the process' stdin. Unless, of course, you're actually concatenating multiple files or maybe a file and stdin together. Disclaimer: I do cat-piping myself quite a bit out of habit, so I'm not trying to look down at the author…

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

I agree with the sentiment, but my critique applies so generally that it must be noted: if a command accepts a filename as a parameter, you should absolutely pass it as a parameter rather than `cat` it over stdin.

For example, you can write this pipeline as:

    grep '^x' foo.txt \
        | sed 's/a/b/g' \
        | awk '{print $2}' \
        | wc -l > bar.txt
This is by no means scientific, but I've got a LaTeX document open right now. A quick `time` says:

    $ time grep 'what' AoC.tex
    real    0m0.045s
    user    0m0.000s
    sys     0m0.000s

    $ time cat AoC.tex | grep what
    real    0m0.092s
    user    0m0.000s
    sys     0m0.047s
Anecdotally, I've witnessed small pipelines that absolutely make sense totally thrash a system because of inappropriate uses of `cat`. When you `cat` a file, the OS must (1) `fork` and `exec`, (2) copy the file to `cat`'s memory, (3) copy the contents of `cat`'s memory to the pipe, and (4) copy the contents of the pipe to `grep`'s memory. That's a whole lot of copying for large files -- especially when the first command grep in the sequence usually performs some major kind of reduction on the input data!

Re: For the Love of Pipes

#234

Pipes are awesome and infuriating. Sometimes they work great -- being able to dump from MySQL into gzip sending across the wire via ssh into gunzip and into my local MySQL without ever touching a file feels nothing short of magic... although the command/incantation to do so took quite a while to finally get right. But far too often they inexplicably fail. For example, I had an issue last year where piping curl to bun…

> Any given combination of pipe tools, there's a kind of random chance they'll actually work in the end or not. While this may be your experience, the mechanism of FIFO pipes in Unix (which is filehandles and buffers, basically), is an old one that is both elegant and robust; it doesn't "randomly" fail due to unreliability of the core algorithm or components. In 20 years, I never had an init script or bash command fa…

If got to agree. I can’t recall a pipe ever failing due to unreliability.

One issue I did used to have (before I discovered ‘-o pipefail’[1]) was the annoyance that if an earlier command in a pipeline failed, all the other commands in the pipeline still ran albeit with no data or garbage data being piped to them.

[1] https://stackoverflow.com/questions/1550933/catching-error-c...

Re: For the Love of Pipes

#235
post #224
post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

She is actually employed at GitHub now. https://twitter.com/jessfraz

Still MSFT, in a way!

Re: For the Love of Pipes

#236
post #172

pipe junkies might like to know about the following tools: * vipe (part of https://joeyh.name/code/moreutils/ - lets you edit text part way through a complex series of piped commands) * pv ( http://www.ivarch.com/programs/pv.shtml - lets you visualise the flow of data through a pipe)

Yes! I love pv. Besides that and tee, can anyone else suggest some more general pipe tools?

http://joeyh.name/code/moreutils/ have a couple more:

* pee: tee standard input to pipes (`pee "some-command" "another-command"`)

* sponge: soak up standard input and write to a file

Though in zsh and bash you can create pee using tee: `tee >(some-command) >(another-command) >/dev/(null`

Re: For the Love of Pipes

#237
post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

> In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`.

I'm guessing you've mentioned using `ls` as a simple, first-thing-that-comes-to-mind example, which is cool. I just wanted to point out that if a person is piping ls's output, there are probably other, far better alternatives, such as (per the link below) `find` and globs:

https://unix.stackexchange.com/a/247973

That has been my experience, at least.

Re: For the Love of Pipes

#238
post #212

Earlier quoted context omitted.

I would hate to see the day HN allowed any way to bold sections of text. It's way more restful purveying a page of uniformly restrained text.

How is that meaningfully different to italics in that regard?

Bold text stands out when visually scanning the page, italics don't.

Re: For the Love of Pipes

#240
post #203

Earlier quoted context omitted.

In your original command, how can 'input' be a program with no arguments?

Oh, damn. You're exactly right. OK, to save some of my face, this will work: grep 'foo' ... at least in zsh and probably bash.

I don’t like that at all. That creates a subshell and is also less readable than

    input | grep foo | sed ...
Post reply on HN