Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

301–310 of 388 posts

Re: The Beauty of Unix Pipelines

#301

Post by rnestler »

cuddlybacon wrote: post #48
cowmix wrote: post #4

I use pipelines as much as the next guy but every time I see post praise how awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on pipelines is pretty spot on too. http://web.mit.edu/~simsong/www/ugh.pdf

I mostly like what they wrote about pipes. I think the example of bloating they talked about in ls at the start of the shell programming section is a good example: if pipelines are so great, why have so many unix utilities felt the need to bloat? I think it a result of there being just a bit too much friction in building a pipeline. A good portion tends to be massaging text formats. The standard unix commands for doi…

> Nushell seems to take a different approach: add structure to command output. By doing that, it seems that a bunch of stuff that is super finicky in the more traditional shells ends up being simple and easy commands with one clear job in nushell. I have never tried it, but it does seem to be movement in the correct direction.

I tried nushell a few times and the commands really compose better due to the structured approach. How would one sort the output of ls by size in bash without letting ls do the sorting? In nushell it is as simple as "ls | sort-by size".

Re: The Beauty of Unix Pipelines

#302

Post by enriquto »

HeadsUpHigh wrote: post #298
enriquto wrote: post #258

Earlier quoted context omitted.

> because spaces are a reasonable thing to put in a filename unless you're on a Unix system. Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.

I'm not going back to CamelCase or underscores for my normal day to day file naming. The problem with spaces only exists inside the IT world and it's something they should find a way around.

I agree. The file load/save dialogs of all GUI should work in such a way that spaces typed by the users in a filename field are always transparently changed to something different (for example, the unicode non-breaking space).

Re: The Beauty of Unix Pipelines

#303

Post by diegocg »

tsimionescu wrote: post #257
gorgoiler wrote: post #47

Earlier quoted context omitted.

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools desig…

The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world. The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.

That's not the fault of pipes, that's the fault of shells like bash. There are shells that deal with spaces and quotes perfectly fine.

Re: The Beauty of Unix Pipelines

#305

Post by chooseaname »

jcranmer wrote: post #38
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them…

All of which are so much easier to deal with than some binary format you have no spec on.

XML? SMH.

Re: The Beauty of Unix Pipelines

#306

Post by majewsky »

benjaminoakes wrote: post #304

I find gron to be much more Unix-y than jq. It "explodes" JSON into single lines for use with grep, sed, etc and can recombine back into JSON as well. https://github.com/TomNomNom/gron

Since jq is sed for JSON, by the transitive property, you're saying that sed is not Unix-y. ;)

Seriously though, I use both, and IMO they serve different purposes. gron is incredibly useful for exploring unknown data formats, especially with any form of

  something | gron | grep something
Once you've figured out how the data format in question works, a jq script is usually more succinct and precise than a chain of gron/{grep,awk,sed,...}/ungron.

So in practice, gron for prompts and jq for scripts.

Re: The Beauty of Unix Pipelines

#307

Post by bregma »

tsimionescu wrote: post #256
gorgoiler wrote: post #196

Earlier quoted context omitted.

Indeed, and the shell lives and breathes spaces. Arrays of arguments are created every time one types a command. Without spaces-as-magic we’d be typing things like this all the time: $ exec(‘ls’, ‘-l’, ‘A B C’) Maybe that’s unrealistic? I mean, if the shell was like that, it probably wouldn’t have exec semantics and would be more like this with direct function calls: $ ls(ls::LONG, ‘A B C’) Maybe we would drop the pa…

I don't think this is the problem people usually complain about. The much bigger problem is that spaces make text-only commands compose badly. $ ls -l `find ./ -name *abc*` Is going to work very nicely if file names have certain properties (no spaces, no special chars), and it's going to break badly if they don't. Quoting is also very simple in simple cases, but explodes in complexity when you add variables and other…

Ah, more problems with shell expansion and substitution.

Pipelines to the rescue. If you used a pipeline instead of shell expansion, it can deal with spaces just peachy.

    $ find . -name "*abc*" -print0 | xargs -0 ls -l
There's no argument that bash (and other shells) make dealing with spaces and quotes troublesome. That has nothing to do with pipelines.

Re: The Beauty of Unix Pipelines

#308

Post by necrotic_comp »

tsimionescu wrote: post #257
gorgoiler wrote: post #47

Earlier quoted context omitted.

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools desig…

The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world. The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.

I've been scripting stuff on the pipeline for over a decade and haven't really run into this much.

You can define a field separator in the command line with the environment variable IFS - i.e. 'IFS=$(echo -en "\n\b");' for newlines - which takes care of the basic cases like spaces in filenames/directory names when doing a for loop, and if I have other highly structured data that is heavily quoted or has some other sort of structure to it, then I either normalize it in some fashion or, as you suggest, write a perl script.

I haven't found it too much of a burden, even when dealing with exceptionally large files.

Re: The Beauty of Unix Pipelines

#309

Post by necrotic_comp »

enriquto wrote: post #258
jcranmer wrote: post #69

Earlier quoted context omitted.

There's a certain irony in responding to criticism of that which you're extolling by saying not to use it. And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.

> because spaces are a reasonable thing to put in a filename unless you're on a Unix system. Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.

At Goldman, the internal language Slang has spaces in variable names. It's insane at first glance and I got into an argument with my manager about why in the world this was acceptable, and he could give me no other answer than "this is the way it is".

But when you realize that space isn't a valid separator between token, seeing things like "Class to Pull Info From Database::Extractor Tool" actually becomes much easier to read and the language becomes highly expressive, helped somewhat by the insane integration into the firm's systems.

I was on your side until I tried it, but it can actually be quite useful, esp. if everything is consistent.

Re: The Beauty of Unix Pipelines

#310

Post by anthk »

naniwaduni wrote: post #155
bananabreakfast wrote: post #140

Earlier quoted context omitted.

Don't you mean Raku? lol

Perl 5 was more suitable for these particular use cases than perl 6 ever tried to be.

Perl5 was and is amazing. Check the articles from https://linuxfocus.com and the old Linux Gazzete magazines from TLDP: https://linuxgazette.net/archives.html.
Post reply on HN