Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

181–190 of 388 posts

Re: The Beauty of Unix Pipelines

#181

Post by antipaul »

A similar philosophy has made the "tidyverse" a much-loved extension of the statistical language R.

Compare the following 2 equivalent snippets. Which one seems more understandable?

    iris_data %>%
        names() %>%
        tolower() %>%
        gsub(".", "_", ., fixed=TRUE) %>%
        paste0("(", ., ")")
or:

    paste0("(", gsub(".", "_", tolower(names(iris_data)), fixed=TRUE), ")")

Re: The Beauty of Unix Pipelines

#183

Post by matheusmoreira »

jiggawatts wrote: post #141
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> Pipes are wonderful! It's wonderful only if compared to worse things, pretending that PowerShell is not a thing, and that Python doesn't exist. UNIX pipes are a stringly-typed legacy that we've inherited from the 1970s. The technical constraints of its past have been internalised by its proponents, and lauded as benefits . To put it most succinctly, the "byte stream" nature of UNIX pipes means that any command that…

> Where each one of those "parse" and "serialise" steps is unique and special, inflexible, and poorly documented.

This can also be a security concern. According to research, a great number of defects with security implications occur at the input handling layers:

http://langsec.org

Re: The Beauty of Unix Pipelines

#184

Post by CountSessine »

viraptor wrote: post #132
coliveira wrote: post #77

Earlier quoted context omitted.

The shell works with spaces, you just need to be careful to quote every file name. The advantage of using file names without spaces is that you can avoid the quotes.

Please don't ever assume you can avoid spaces and quotes. It's just a time bomb. It's like people saying you don't need to escape SQL values because they come from constants. Yes, they do... today. It's not just quoting either. It's setting the separator value and reverting it correctly. It's making sure you're still correct when you're in a function in an undefined state. It's a lot of overhead in larger projects.

Sure, but in a larger project your won’t be coding in shell script, right?

Re: The Beauty of Unix Pipelines

#185

Post by yjftsjthsd-h »

shrimp_emoji wrote: post #91
gorgoiler wrote: post #33

Earlier quoted context omitted.

Exactly. “ Linux, ew !” A big part of teaching computer science to children is breaking this obsession with approaching a computer from the top down — the old ICT ways, and the love of apps — and learning that it is a machine under your own control the understanding of which is entirely tractable from the bottom up. Unlike the natural sciences, computer science (like math) is entirely man made, to its advantage. No m…

My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin…

- Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion)

It has its pain points, but the things that (ba)sh is good at, it's really good at and python, in my experience, doesn't compete. Dumb example: `tar czf shorttexts.tar.gz $(find . -type f | grep ^./...\.txt) && rsync shorttexts.tar.gz laptop:`. I could probably make a python script that did the equivalent, but it would not be a one-liner, and my feeling is that it would be a lot uglier.

- Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin? multiple users on one computer?? user groups??? entering sudo password all the time????)

In order: /usr/bin is for packaged software; /usr/local is the sysadmin's local builds. Our servers at work have a great many users, and we're quite happy with it. I... have no idea why you would object to groups. If you're entering your sudo password all the time, you're probably doing something wrong, but you're welcome to tell it to not require a password or increase the time before reprompting.

- The design blunder of assuming that people will read insanely dense man pages (the amount of StackOverflow questions[0][1][2] that exist for anything that can be located in documentation are a testament to this)

I'll concede that many GNU/Linux manpages are a bit on the long side (hence bro and tldr pages), but having an actual manual, and having it locally (works offline) is quite nice. Besides which, you can usually just search through it and find what you want.

- And of course no bideo gambes and other things (although hopefully webapps will liberate us from OS dependence soon[3][4])

Webapps have certainly helped the application situation, but Steam and such have also gotten way better; it's moved from "barely any video games on Linux" to "a middling amount of games on Linux".

Re: The Beauty of Unix Pipelines

#186

Post by pwdisswordfish2 »

ketanmaheshwari wrote: post #6

Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipeline is a process and a debugging overhead in case something goes wrong. A case in point is this pipeline that I came across in the wild: TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep default | cut -f1 -d ' ') | grep…

The -E on the grep makes no sense. I do not see any ERE. Looks like everything here could be done with a single invocation of sed. Can anyone share some sample output of kubectl?

Re: The Beauty of Unix Pipelines

#187

Post by yjftsjthsd-h »

ryandrake wrote: post #104
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> (2) everything (ish) is a file I'm not all that knowledgable about Unix history, but one thing that has always puzzled me was that for whatever reason network connections (generally) aren't files. While I can do: cat to read from a serial device, I've always wondered why I can't do something like: bind /tmp/mysocket 12.34.56.78 80 cat It is weird that so many things in Unix are somehow twisted into files (/dev/fb0?…

Sounds like you want netcat?

Re: The Beauty of Unix Pipelines

#188

Post by fomine3 »

atombender wrote: post #149

Pipes are a great idea, but are severely hampered by the many edge cases around escaping, quoting, and, my pet peeve, error handling. By default, in modern shells, this will actually succeed with no error: $ alias fail=exit 1 $ find / | fail | wc -l; echo $? 0 0 You can turn on the "pipefail" option to remedy this: $ set -o pipefail $ find / | fail | wc -l; echo $? 0 1 Most scripts don't, because the option makes eve…

set -e makes another pain for command that nonzero isn't mean failed (ex. diff). It changes semantics for whole script.

Re: The Beauty of Unix Pipelines

#189

Post by Shared404 »

gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

>Unix is seriously uncool with young people...

Not all of us. I much prefer the Unix way of doing things, it just makes more sense then just trying to become another Windows.

Re: The Beauty of Unix Pipelines

#190

Post by atombender »

fomine3 wrote: post #188
atombender wrote: post #149

Pipes are a great idea, but are severely hampered by the many edge cases around escaping, quoting, and, my pet peeve, error handling. By default, in modern shells, this will actually succeed with no error: $ alias fail=exit 1 $ find / | fail | wc -l; echo $? 0 0 You can turn on the "pipefail" option to remedy this: $ set -o pipefail $ find / | fail | wc -l; echo $? 0 1 Most scripts don't, because the option makes eve…

set -e makes another pain for command that nonzero isn't mean failed (ex. diff). It changes semantics for whole script.

If by pain you mean you have to handle errors, yes, that's what you have to do. It's no different from checking the return code of functions in C.
Post reply on HN