Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

11–20 of 388 posts

Re: The Beauty of Unix Pipelines

#11
post #5

I love pipelines. I don't know the elaborate sublanguages of find, awk, and others, to exploit them adequately. I also love Python, and would rather use Python than those sublanguages. I'm developing a shell based on these ideas: https://github.com/geophile/marcel .

Your project looks really cool.

I am pretty sure I've seen a Python-based interactive shell a few years ago but I can't remember the name. Have you heard of it?

Re: The Beauty of Unix Pipelines

#12

I have problems understanding what commands I can pipe. Some work some don't.

The commands that read their input from standard input could be used as receiver in a pipeline. For instance: `A | B` is a short form of `A > tmp.txt; B There are some commands that need literal arguments which is different than standard input. For instance the echo command. `echo < list.txt` will not work assuming you want to print the items inside the list. `echo itemA itemB itemC` will work. This is where xargs comes into play -- it converts the standard input stream to literal arguments among other things.

Re: The Beauty of Unix Pipelines

#13
post #11
post #5

I love pipelines. I don't know the elaborate sublanguages of find, awk, and others, to exploit them adequately. I also love Python, and would rather use Python than those sublanguages. I'm developing a shell based on these ideas: https://github.com/geophile/marcel .

Your project looks really cool. I am pretty sure I've seen a Python-based interactive shell a few years ago but I can't remember the name. Have you heard of it?

Do you mean IPython? My understanding is that IPython more of a REPL for Python, less of a shell.

Re: The Beauty of Unix Pipelines

#14
post #4

I use pipelines as much as the next guy but every time I see post praise how awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on pipelines is pretty spot on too. http://web.mit.edu/~simsong/www/ugh.pdf

I read the introduction. Sounds like Norman would have liked to hear about Plan 9.

Re: The Beauty of Unix Pipelines

#15
post #11
post #5

I love pipelines. I don't know the elaborate sublanguages of find, awk, and others, to exploit them adequately. I also love Python, and would rather use Python than those sublanguages. I'm developing a shell based on these ideas: https://github.com/geophile/marcel .

Your project looks really cool. I am pretty sure I've seen a Python-based interactive shell a few years ago but I can't remember the name. Have you heard of it?

I imagine you are thinking of xonsh? https://xon.sh/

Re: The Beauty of Unix Pipelines

#16
post #11
post #5

I love pipelines. I don't know the elaborate sublanguages of find, awk, and others, to exploit them adequately. I also love Python, and would rather use Python than those sublanguages. I'm developing a shell based on these ideas: https://github.com/geophile/marcel .

Your project looks really cool. I am pretty sure I've seen a Python-based interactive shell a few years ago but I can't remember the name. Have you heard of it?

Perhaps you are thinking of xonsh https://xon.sh/

Edit: x1798DE beat me to it :D

Re: The Beauty of Unix Pipelines

#17
post #14
post #4

I use pipelines as much as the next guy but every time I see post praise how awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on pipelines is pretty spot on too. http://web.mit.edu/~simsong/www/ugh.pdf

I read the introduction. Sounds like Norman would have liked to hear about Plan 9.

I have a hard time believing he would have been unfamiliar with Plan 9. It wasn’t exactly obscure in the research community at the time. See the USENIX proceedings in the late 80s and 90s.

This is mere speculation, but I doubt he would have appreciated Plan 9.

Re: The Beauty of Unix Pipelines

#19
Powershell is the ultimate expression of the Unix pipeline imo.

Passing objects through the pipeline and being able to access this data without awk/sed incantations is a blessing for me.

I think anyone who appreciates shell pipelines and python can grok the advantages of the approach taken by Powershell, in a large way it is directly built upon existing an Unix heritage.

I'm not so good at explaining why, but for anyone curious please have a look at the Monad manifesto by Jeffrey Snover

https://devblogs.microsoft.com/powershell/monad-manifesto-th...

You may not agree with the implementation, but the ideas being it, I think, are worth considering.

Re: The Beauty of Unix Pipelines

#20

Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipeline is a process and a debugging overhead in case something goes wrong. A case in point is this pipeline that I came across in the wild: TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep default | cut -f1 -d ' ') | grep…

I think people can go through a few stages of their shell-foo.

The first involves a lot of single commands and temporary files.

The second uses pipes, but only tacks on commands with no refactoring.

The third would recognize that all the grep and cut should just be awk, that you can redirect the cumulative output of a control statement, that subprocesses and coroutines are your friend. We should all aspire to this.

The fourth stage is rare. Some people start using weird file descriptors to chuck data around pipes, extensive process substitution, etc. This is the Perl of shell and should be avoided. The enlightened return back to terse readable scripts, but with greater wisdom.

Also, I always hear of people hating awk and sed and preferring, say, python. Understanding awk and especially sed will make your python better. There really is a significant niche where the classic tools are a better fit than a full programming language.

Post reply on HN