Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

251–260 of 388 posts

Re: The Beauty of Unix Pipelines

#251

Post by philsnow »

ryandrake wrote: post #104
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> (2) everything (ish) is a file I'm not all that knowledgable about Unix history, but one thing that has always puzzled me was that for whatever reason network connections (generally) aren't files. While I can do: cat to read from a serial device, I've always wondered why I can't do something like: bind /tmp/mysocket 12.34.56.78 80 cat It is weird that so many things in Unix are somehow twisted into files (/dev/fb0?…

elsewhere in this thread somebody mentions socat, but you can do it entirely within bash. from https://www.linuxjournal.com/content/more-using-bashs-built-... :

    exec 3/dev/tcp/www.google.com/80
    echo -e "GET / HTTP/1.1\r\nhost: http://www.google.com\r\nConnection: close\r\n\r\n" >&3
    cat
https://news.ycombinator.com/item?id=23422423 has a good point that "everything is a file" is maybe less useful than "everything is a file descriptor". the shell is a tool for setting up pipelines of processes and linking their file descriptors together.

Re: The Beauty of Unix Pipelines

#252

Post by enriquto »

marcthe12 wrote: post #247
enriquto wrote: post #205

Earlier quoted context omitted.

This is not about "code golfing", the "sort|uniq -c|sort" combo is in the hall of fame of great code lines. The PATH thing in my script was an unnecessary distractor, consider this: file -b /bin/* /usr/bin/* |cut -d' ' -f-2|sort|uniq -c|sort -n There are no bizarre escapes nor anything. Besides, the "file" program is called only once. The python equivalent that you wrote may be better if you want to store it somewher…

find-xargs combo could have been better. It also prevents the case where you cross arg max

Definitely! It is also much slower, though, especially if you need to add "-L 1". I prefer to try first that way, if the argument list is too long, then try xargs.

On the other hand, the limitations on the command lenght of modern shells are idiotic. The only limitation should be the amount of available RAM, not an artificially imposed limit.

Re: The Beauty of Unix Pipelines

#254

Post by JdeBP »

DSMan195276 wrote: post #134
imglorp wrote: post #110

Earlier quoted context omitted.

That's correct, which is why I said _starts_ with the streams. The child can do what it wants with them after that.

His point was that if you close stdin/stdout/stderr and then fork a child, the new child will not have all of those streams when it starts - the three streams are a convention, not something enforced by the kernel when creating a new process. That said, it's still a bit of a pedantic point, you can expect to always have those three fds in the same way you can expect `argv[0]` to represent the name of your executable…

Actually, it's not a pedantic point. It's a security flaw waiting to happen (as it actually has) if one assumes that one always has three open file descriptors.

The privileged program opens a sensitive data file for writing, it gets assigned (say) file descriptor 2, because someone in a parent process took some advice to "close your standard I/O because you are a dæmon" to heart, other completely unrelated library code somewhere else blithely logs network-supplied input to standard error without sanitizing it, and (lo!) there's a remotely accessible file overwrite exploit of the sensitive data file.

Ironically, the correct thing to do as a dæmon is to not close standard I/O, but to leave setting it up to the service management subsystem, and use it as-is. That way, when the service management subsystem connects standard output and error to a pipe and arranges to log everything coming out the other end, daemontools-style, the idea actually works. (-:

Re: The Beauty of Unix Pipelines

#255

Post by tsimionescu »

coliveira wrote: post #77
jcranmer wrote: post #69

Earlier quoted context omitted.

There's a certain irony in responding to criticism of that which you're extolling by saying not to use it. And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.

The shell works with spaces, you just need to be careful to quote every file name. The advantage of using file names without spaces is that you can avoid the quotes.

The shell (bash, anyway) has a weird mix of support and lack of support for spaces/lists/etc.

For example, there is no way to store the `a b "c d"` in a regular variable in a way where you can then call something similar to `ls $VAR` and get the equivalent of `ls a b "c d"`. You can either get the behavior of `ls a b c d` or `ls "a b c d"`, but if you need `ls a b "c d"` you must go for an array variable with new syntax. This isn't necessarily a big hurdle, but it indicates that the concepts are hard to grasp and possibly inconsistent.

Re: The Beauty of Unix Pipelines

#256

Post by tsimionescu »

gorgoiler wrote: post #196
coliveira wrote: post #77

Earlier quoted context omitted.

The shell works with spaces, you just need to be careful to quote every file name. The advantage of using file names without spaces is that you can avoid the quotes.

Indeed, and the shell lives and breathes spaces. Arrays of arguments are created every time one types a command. Without spaces-as-magic we’d be typing things like this all the time: $ exec(‘ls’, ‘-l’, ‘A B C’) Maybe that’s unrealistic? I mean, if the shell was like that, it probably wouldn’t have exec semantics and would be more like this with direct function calls: $ ls(ls::LONG, ‘A B C’) Maybe we would drop the pa…

I don't think this is the problem people usually complain about.

The much bigger problem is that spaces make text-only commands compose badly.

    $ ls -l `find ./ -name *abc*`
Is going to work very nicely if file names have certain properties (no spaces, no special chars), and it's going to break badly if they don't. Quoting is also very simple in simple cases, but explodes in complexity when you add variables and other commands into the mix.

Re: The Beauty of Unix Pipelines

#257

Post by tsimionescu »

gorgoiler wrote: post #47
jcranmer wrote: post #38

Earlier quoted context omitted.

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them…

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools desig…

The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world.

The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.

Re: The Beauty of Unix Pipelines

#258

Post by enriquto »

jcranmer wrote: post #69
gorgoiler wrote: post #47

Earlier quoted context omitted.

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools desig…

There's a certain irony in responding to criticism of that which you're extolling by saying not to use it. And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.

> because spaces are a reasonable thing to put in a filename unless you're on a Unix system.

Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.

Re: The Beauty of Unix Pipelines

#259

Post by outadoc »

enriquto wrote: post #258
jcranmer wrote: post #69

Earlier quoted context omitted.

There's a certain irony in responding to criticism of that which you're extolling by saying not to use it. And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.

> because spaces are a reasonable thing to put in a filename unless you're on a Unix system. Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.

You should take a stroll in the real world sometime, where spaces and Unicode exists :)

Re: The Beauty of Unix Pipelines

#260

Post by RMPR »

cryptonector wrote: post #62
marcosdumay wrote: post #35

Earlier quoted context omitted.

Hum... Those other abstractions on that section are great and everything, but it really sucks to write them interactively. Pipelines also lack some concept similar to exceptions, but it would also suck to handle those interactively.

set -o pipefail

Let's be more hardcore

    set -euo pipefail
Post reply on HN