Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

21–30 of 388 posts

Re: The Beauty of Unix Pipelines

#21

Post by gorgoiler »

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix:

(1) everything is text

(2) everything (ish) is a file

(3) including pipes and fds

(4) every piece of software is accessible as a file, invoked at the command line

(5) ...with local arguments

(6) ...and persistent globals in the environment

A lot of understanding comes once you know what execve does, though such knowledge is of course not necessary. It just helps.

Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material.

Re: The Beauty of Unix Pipelines

#22

Post by m0xte »

ketanmaheshwari wrote: post #6

Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipeline is a process and a debugging overhead in case something goes wrong. A case in point is this pipeline that I came across in the wild: TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep default | cut -f1 -d ' ') | grep…

Every time I see that sort of stuff I end up | perl, still after 20 odd years. Sometimes i fine perl is not installed and I die a little.

Re: The Beauty of Unix Pipelines

#23

Post by young_unixer »

salmo wrote: post #20
ketanmaheshwari wrote: post #6

Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipeline is a process and a debugging overhead in case something goes wrong. A case in point is this pipeline that I came across in the wild: TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep default | cut -f1 -d ' ') | grep…

I think people can go through a few stages of their shell-foo. The first involves a lot of single commands and temporary files. The second uses pipes, but only tacks on commands with no refactoring. The third would recognize that all the grep and cut should just be awk, that you can redirect the cumulative output of a control statement, that subprocesses and coroutines are your friend. We should all aspire to this. T…

My most complex use of the shell is defining aliases in .bashrc and have never felt the need to go further. Do you recommend learning all of that for someone like me?

If so, what resource do you recommend?

Re: The Beauty of Unix Pipelines

#24

Post by khimaros »

geophile wrote: post #5

I love pipelines. I don't know the elaborate sublanguages of find, awk, and others, to exploit them adequately. I also love Python, and would rather use Python than those sublanguages. I'm developing a shell based on these ideas: https://github.com/geophile/marcel .

The lispers/schemers in the audience may be interested in Rash https://docs.racket-lang.org/rash/index.html which lets you combine an sh-like language with any other Racket syntax.

Re: The Beauty of Unix Pipelines

#26

Post by fenwick67 »

cowmix wrote: post #4

I use pipelines as much as the next guy but every time I see post praise how awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on pipelines is pretty spot on too. http://web.mit.edu/~simsong/www/ugh.pdf

From the intro:

> As for me? I switched to the Mac.

It's amusing that Apple would go on to switch to a unix-based OS in '01.

Re: The Beauty of Unix Pipelines

#27

Post by ianai »

gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

How’s it uncool? Just not “ mobile”?

Re: The Beauty of Unix Pipelines

#28

Post by lordleft »

ketanmaheshwari wrote: post #6

Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipeline is a process and a debugging overhead in case something goes wrong. A case in point is this pipeline that I came across in the wild: TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep default | cut -f1 -d ' ') | grep…

A little bit of grep / awk goes very, very far.

Re: The Beauty of Unix Pipelines

#29

Post by mehrdadn »

What kills me about pipelines is when I pipe into xargs and then suddenly can't kill things properly with Ctrl+C. Often I have to jump through hoops to parse arguments into arrays and avoid xargs just for this. (This has to do with stdin being redirected. I don't recall if there's anything particular about xargs here, but that's where it usually comes up.)

Re: The Beauty of Unix Pipelines

#30

Post by laumars »

gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> everything is text

Everything is a byte stream. Usually that means text but sometimes it doesn't. Which means you can do fun stuff like:

- copy file systems over a network: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm...

- stream a file into gzip

- backup or restore an SD card using `cat`

Post reply on HN