Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For the Love of Pipes

blog.jessfraz.com

Post reply on HN
Print view View original

221–230 of 323 posts

Re: For the Love of Pipes

#221

Post by sevensor »

nojvek wrote: post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

If you're using cut and awk to get a field out of ls, maybe what you want is actually stat(1)?

Re: For the Love of Pipes

#222

Post by crazygringo »

EE84M3i wrote: post #215
crazygringo wrote: post #198

Pipes are awesome and infuriating. Sometimes they work great -- being able to dump from MySQL into gzip sending across the wire via ssh into gunzip and into my local MySQL without ever touching a file feels nothing short of magic... although the command/incantation to do so took quite a while to finally get right. But far too often they inexplicably fail. For example, I had an issue last year where piping curl to bun…

Perhaps your example was contrived, but why would you pipe into gzip instead of using transparent ssh compression?

Because it simply never occurred to me to check if ssh would have compression built-in.

Because why would it? If the UNIX philosophy is to separate out tools and pipe them, then the UNIX philosophy should be to pipe through gzip and gunzip, not for ssh to provide its own redundant compression option, right?

Re: For the Love of Pipes

#223

Post by masklinn »

beachy wrote: post #212
masklinn wrote: post #196

Earlier quoted context omitted.

That would break actual code snippet. What would solve most of the problems is HN actually implementing markdown instead of the current half-assed crap.

I would hate to see the day HN allowed any way to bold sections of text. It's way more restful purveying a page of uniformly restrained text.

> I would hate to see the day HN allowed any way to bold sections of text.

HN already has shitty italics (shitty in that it commonly matches and eats things you don't want to be italicised e.g. multiplications, pointers, … in part though not only because HN doesn't have inline code). "bold" can just be styled as italics, or it can be styled as a medium or semibold. It's not an issue, and even less worth it given how absolute garbage the current markup situation is.

Re: For the Love of Pipes

#224

Post by nijaru »

nojvek wrote: post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

She is actually employed at GitHub now.

https://twitter.com/jessfraz

Re: For the Love of Pipes

#225

Post by acdha »

crazygringo wrote: post #222
EE84M3i wrote: post #215

Earlier quoted context omitted.

Perhaps your example was contrived, but why would you pipe into gzip instead of using transparent ssh compression?

Because it simply never occurred to me to check if ssh would have compression built-in. Because why would it? If the UNIX philosophy is to separate out tools and pipe them, then the UNIX philosophy should be to pipe through gzip and gunzip, not for ssh to provide its own redundant compression option, right?

This is a good example of where that simple rule breaks down: piping it would only work when you are running a command and feeding its output to a different location whereas having it in SSH helps with everything so e.g. typing `ls` in a big directory, `cat`-ing a file, using `scp` on text, etc. benefits.

Re: For the Love of Pipes

#226

Post by invsblduck »

crazygringo wrote: post #198

Pipes are awesome and infuriating. Sometimes they work great -- being able to dump from MySQL into gzip sending across the wire via ssh into gunzip and into my local MySQL without ever touching a file feels nothing short of magic... although the command/incantation to do so took quite a while to finally get right. But far too often they inexplicably fail. For example, I had an issue last year where piping curl to bun…

> Any given combination of pipe tools, there's a kind of random chance they'll actually work in the end or not.

While this may be your experience, the mechanism of FIFO pipes in Unix (which is filehandles and buffers, basically), is an old one that is both elegant and robust; it doesn't "randomly" fail due to unreliability of the core algorithm or components. In 20 years, I never had an init script or bash command fail due to the pipe(3) call itself being unreliable.

If you misunderstand detailed behavior of the commands you are stitching together--or details of how you're transiting the network in case of an ssh remote command--then yes, things may go wrong. Especially if you are creating Hail Mary one-liners, which become unwieldy.

Re: For the Love of Pipes

#227

Post by laumars »

zwp wrote: post #195
laumars wrote: post #92

Earlier quoted context omitted.

I'm not going to argue that UNIX got everything right because I don't believe that to be the case either but I don't agree with those specific points: > This means that you can’t send an object and the code for the class definition necessary to implement the object. To some degree you can and I do just this with my own shell I've written. You just have to ensure that both ends of the pipe understands what is being se…

>> You can’t send file handles > Actually that's exactly how piping works Also SCM_RIGHTS, which exists exactly for this purpose (see cmsg(3), unix(7) or https://blog.cloudflare.com/know-your-scm_rights/ for a gentler introduction and application). That's been around since BSD 4.3, which predates the Hater's Handbook 1ed by 4 years or so.

Yeah I had mentioned UNIX domain sockets. However your post does add a lot of good detail on them which I had left off.

Re: For the Love of Pipes

#228

Post by invsblduck »

yaakushi wrote: post #85

I'm probably nitpicking, but if you're using cat to pipe a single file into the sdtin of another program, you most likely don't need the cat in the first place, you can just redirect the file to the process' stdin. Unless, of course, you're actually concatenating multiple files or maybe a file and stdin together. Disclaimer: I do cat-piping myself quite a bit out of habit, so I'm not trying to look down at the author…

Not nitpicking. Useless Use of Cat is an old thing: http://catb.org/jargon/html/U/UUOC.html

Re: For the Love of Pipes

#229

Post by sideeffffect »

VladimirGolovin wrote: post #110
benj111 wrote: post #10

Why isn't the pipe a construct that has caught on in 'proper' languages?

It has, in the form of function composition, as other replies show. However, the Unix pipe demonstrates a more interesting idea: composable programs on the level of the OS. Nowadays, most of the user-facing desktop programs have GUIs, so the 'pipe' operator that composes programs is the user himself. Users compose programs by saving files from one program and opening them in another. The data being 'piped' through su…

maybe I'm being overly pedantic, but people seem to be confused about this:

the pipes in your typical functional language (`|>`) is not a form of function composition, like

```

f >> g === x -> g(f(x))

```

but function application, like

```

f x |> g === g(f(x))

x |> f |> g // also works, has the same meaning

f |> g // just doesn't work, sorry :(

```

Re: For the Love of Pipes

#230

Post by _whiteCaps_ »

foreigner wrote: post #189
cmsj wrote: post #172

pipe junkies might like to know about the following tools: * vipe (part of https://joeyh.name/code/moreutils/ - lets you edit text part way through a complex series of piped commands) * pv ( http://www.ivarch.com/programs/pv.shtml - lets you visualise the flow of data through a pipe)

Yes! I love pv. Besides that and tee, can anyone else suggest some more general pipe tools?

pee

https://linux.die.net/man/1/pee

Post reply on HN