Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

151–160 of 388 posts

Re: The Beauty of Unix Pipelines

#151

Post by mjevans »

dredmorbius wrote: post #131
jrumbut wrote: post #84

Earlier quoted context omitted.

Also dd, which may be the disk destroyer but is also a great tool for binary file miracles. See one here: https://unix.stackexchange.com/questions/6852/best-way-to-re...

"data definition", from MVS JCL. https://news.ycombinator.com/user?id=dredmorbius

I recall hearing conjecture that cc was already a routine, and that "copy and convert" became dd as a result in a really old system that may have been pre-unix.

Researching, it appears that there are multiple attributions, because it's just that obvious and necessary a routine.

https://www2.cs.duke.edu/csl/docs/unix_course/intro-98.html

https://unix.stackexchange.com/questions/6804/what-does-dd-s...

https://en.wikipedia.org/wiki/Dd_(Unix)

Re: The Beauty of Unix Pipelines

#152

Post by Koshkin »

pwdisswordfish2 wrote: post #144
gorgoiler wrote: post #47

Earlier quoted context omitted.

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools desig…

[C] Long lived tools one throws together oneself Stable tools designed for oneself

The unknown knowns?

Re: The Beauty of Unix Pipelines

#153

Post by Koshkin »

cobalt wrote: post #139
ssivark wrote: post #75

Earlier quoted context omitted.

Serialized bytestreams do compose better than graphical applications. But that is setting a very low bar. For example, allowing passing around dicts/maps/json (and possibly other data structures) would already be a massive improvement. You know what might be even better — passing around objects you could interact with by passing messages (gasp! cue, Alan Kay). While posix and streams are nice (if you squint, files lo…

try powershell, its on linux too

Indeed, if it wasn’t for its reliance on .NET, it looks like PowerShell could be a massive improvement over Unix shell.

Re: The Beauty of Unix Pipelines

#154

Post by Koshkin »

matheusmoreira wrote: post #76
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

1. All data is bytes 2. Everything is a file descriptor 3. File descriptors are things that support standard I/O system calls such as read and write

You are describing Plan 9.

Re: The Beauty of Unix Pipelines

#155

Post by naniwaduni »

bananabreakfast wrote: post #140
anthk wrote: post #95

Earlier quoted context omitted.

That's Perl role, not Python. You are like 20 years late, kid. Have fun with that shitty whitespace syntax. At least Perl can be refactorized.

Don't you mean Raku? lol

Perl 5 was more suitable for these particular use cases than perl 6 ever tried to be.

Re: The Beauty of Unix Pipelines

#157

Post by codemac »

atombender wrote: post #149

Pipes are a great idea, but are severely hampered by the many edge cases around escaping, quoting, and, my pet peeve, error handling. By default, in modern shells, this will actually succeed with no error: $ alias fail=exit 1 $ find / | fail | wc -l; echo $? 0 0 You can turn on the "pipefail" option to remedy this: $ set -o pipefail $ find / | fail | wc -l; echo $? 0 1 Most scripts don't, because the option makes eve…

Just use a better shell. rc handles this wonderfully, $? is actually called $status, and it's an array, depending on the number of pipes.

Re: The Beauty of Unix Pipelines

#158

Post by zamalek »

shakna wrote: post #150
zamalek wrote: post #101

Earlier quoted context omitted.

I am firmly in the "ugh" camp. I strongly suspect that the fawning that occurs over pipelines is because of sentimentality more than practicality. Extracting information from text using a regular expression is fragile. Writing fragile code brings me absolutely no joy as a programmer - unless I am trying to flex my regex skills. If you really look at how pipelines are typically used is: lines are analogous to objects…

If you're having to extract your data using a regex, then the data probably isn't well-formed enough for a shell pipeline. It's doable, but a bad idea. Regex should not be the first hammer you reach for, because it's a scalpel. I recently wanted cpu cores + 1. That could be a single regex. But this is more maintainable, and readable: echo '1 + '"$(grep 'cpu cores' /proc/cpuinfo | tail -n1 | awk -F ':' '{print $2}')"…

Would anyone be able to approach this, without knowledge of /proc/cpuinfo, and understand what `$2` refers to?

> Regex should not be the first hammer you reach for, because it's a scalpel.

You used a regex tool: grep.

Re: The Beauty of Unix Pipelines

#159

Post by russellbeattie »

mongol wrote: post #98
russellbeattie wrote: post #82

I think there's an interesting inflection point between piping different utilities together to get something done, and just whipping up a script to do the same thing instead. First I'll use the command line to, say, grab a file from a URL, parse, sort and format it. If I find myself doing the same commands a lot, I'll make a .sh file and pop the commands in there. But then there's that next step, which is where Bash…

I have never come across a NodeJS script instead of a shell script. Can you suggest an example to look at that shows the advantages of this?

Here's a JS script [1] I wrote a little while ago just for my own use that queries the CDC for the latest virus numbers, then calcs the average, formats the data and prints to the command line. You can pass in a number of days, otherwise it pulls the last 14 days.

$ node query-cdc.js 7

It's nothing special, but I wouldn't want to try to do this with command line utilities. (And yes, it was a bit uglier, but I cleaned it up of random crap I had thrown in before posting it.)

1. https://gist.github.com/russellbeattie/f9edf91115b43d6d7ca3c...

Post reply on HN