Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

201–210 of 388 posts

Re: The Beauty of Unix Pipelines

#201

Post by gorgoiler »

jiggawatts wrote: post #141
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> Pipes are wonderful! It's wonderful only if compared to worse things, pretending that PowerShell is not a thing, and that Python doesn't exist. UNIX pipes are a stringly-typed legacy that we've inherited from the 1970s. The technical constraints of its past have been internalised by its proponents, and lauded as benefits . To put it most succinctly, the "byte stream" nature of UNIX pipes means that any command that…

Typed streams sound great.

Most data being line oriented (with white space field separators) has historically worked well enough, but I get your point that the wheels will eventually fall off.

It’s important to remember that the shell hasn’t always been about high concept programming tasks.

  $ grep TODO xmas-gifts
  grandma TODO
  auntie Sian TODO
  jeanie MASTODON T shirt
The bug (bugs!) in the above example doesn’t really matter in the context of the task at hand.

Re: The Beauty of Unix Pipelines

#203

Post by gorgoiler »

quickthrower2 wrote: post #87
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

Can you give me any clue as to what execve does? I looked at the man page but none the wiser. Sounds like magic from what I read there. I'm from a Windows backgroud and not used to pipes.

Its how you launch (execute) a new program.

Such programs typically don’t have any interaction so the v and the e parts refer to the two ways you can control what the program does.

v: a vector (list) of input parameters (aka arguments)

e: an environment of key-value pairs

The executed program can interpret these two in any way it wishes to, though there are many conventions that are followed (and bucked by the contrarians!) like using “-“ to prefix flags/options.

This is in addition to any data sent to the program’s standard input — hence the original discussion about using pipes to send the output of one command to the input of another.

Re: The Beauty of Unix Pipelines

#204

Post by enriquto »

fomine3 wrote: post #175
jcranmer wrote: post #38

Earlier quoted context omitted.

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them…

The fact that filename can contain anything except NULL/slash is really pain. I often write a shell script that treats LF as separator but I know it's not good.

filenames are variable names. Choosing good filenames is the first step in writing a nice shell script in a situation that you control.

Re: The Beauty of Unix Pipelines

#205

Post by enriquto »

viraptor wrote: post #126
enriquto wrote: post #66

If you eval this simple pipeline file -b `echo $PATH:|sed 's/:/\/* /g'`|cut -d\ -f-2|sort|uniq -c|sort -n it prints a histogram of the types of all the programs on your path (e.g., whether they are shell, python, perl scripts or executable binaries). How can you ever write such a cute thing in e.g., python or, god forbid, java?

You don't. If you use it once, meh. If you share it with anyone or preserve for the future, why would you want it to be cute? It's just a few lines in python, probably takes just as long to write because you don't have to play with what needs to be escaped and what doesn't. You can actually tell what's the intent of each line and it doesn't fail on paths starting with minuses or including spaces. Outside of one time…

This is not about "code golfing", the "sort|uniq -c|sort" combo is in the hall of fame of great code lines. The PATH thing in my script was an unnecessary distractor, consider this:

    file -b /bin/* /usr/bin/* |cut -d' ' -f-2|sort|uniq -c|sort -n
There are no bizarre escapes nor anything. Besides, the "file" program is called only once. The python equivalent that you wrote may be better if you want to store it somewhere, but it takes a lot to type, and it serves a different purpose. The shell line is something that you write once and run it because it falls naturally from your fingers. Moreover, you can run it everywhere, as it works in bash, zsh, pdksh and any old shell. The python version requires an appropriate python version installed (3 not 2).

Re: The Beauty of Unix Pipelines

#206

Post by enriquto »

nojito wrote: post #108
jpxw wrote: post #83

Earlier quoted context omitted.

grep, awk, sed, cut, sort, uniq and join are the Swiss Army knife of working with tabulated data on the command line.

And yet they are absolutely useless if columns get swapped silently.

Everything breaks down if columns are swapped silently (or database records are renamed silently). What do you mean?

Re: The Beauty of Unix Pipelines

#207

Post by enriquto »

m0xte wrote: post #22
ketanmaheshwari wrote: post #6

Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipeline is a process and a debugging overhead in case something goes wrong. A case in point is this pipeline that I came across in the wild: TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep default | cut -f1 -d ' ') | grep…

Every time I see that sort of stuff I end up | perl, still after 20 odd years. Sometimes i fine perl is not installed and I die a little.

In what unix system perl is not installed? It is in the default install on current versions of macos (where it comprises the majority of interpreted programs), openbsd, freebsd, ubuntu and fedora linux.

Re: The Beauty of Unix Pipelines

#208

Post by gorgoiler »

asdff wrote: post #148
kccqzy wrote: post #99

Earlier quoted context omitted.

It replaces the executable in the current process with a different executable. It's kind of like spawning a new process with an executable, except it's the same PID, and file descriptors without CLOEXEC remain open in their original state.

whats a simple example of using this?

It’s how all new processes in Unix are created. There is no “spawn new process running this code” operation. Instead:

(1) you duplicate yourself with fork()

(2) the parent process continues about its business

(3) the created process calls execve() to become something else

This is the mechanism by which all new processes are created, from init (process 1) to anything you run from your shell.

Real life examples here, in this example of a Unix shell:

https://github.com/mirror/busybox/blob/master/shell/ash.c

Re: The Beauty of Unix Pipelines

#209

Post by mturmon »

salmo wrote: post #81
young_unixer wrote: post #23

Earlier quoted context omitted.

My most complex use of the shell is defining aliases in .bashrc and have never felt the need to go further. Do you recommend learning all of that for someone like me? If so, what resource do you recommend?

I may be a bad person to ask. I like history a lot. A lot of this stuff was originally picked up by secretaries and non-computer people working on manuals for Bell Labs. If you understand it, it will save you time. It will also make you use and understand Linux/Unix better, which will bleed into any other programming you do (assuming you write other code). The skill also comes up a lot in strange places like Dockerfi…

You’ve left some good tips. sed, awk, and grep. getopts. Read the man pages. Try to automate even simple series of commands.

Don’t be shy about running a whole pipeline of commands (in an automation script) just to isolate one field within one line of one file and add one to it, and store the result in a shell variable.

Re: The Beauty of Unix Pipelines

#210

Post by enriquto »

blacksqr wrote: post #120
dataflow wrote: post #85

Earlier quoted context omitted.

Try doing that in e.g. Make

In the early 2k's I discovered that it was impossible to quote or escape spaces in filepaths in a makefile. I naively reported it to the Make maintainer as a bug. The maintainer responded by saying, basically, that the behavior was deeply baked in and was not going to change.

This is a nice feature of make. People who put spaces on filenames are justly punished by not being given the goodness of make.
Post reply on HN