The Beauty of Unix Pipelines • Hacker News Board

Re: The Beauty of Unix Pipelines

#211

Post by enriquto » Fri, Jun 05, 2020, 4:56 AM UTC

sedatk wrote: post #18

It's ironic that the article ends with Python code. You could have done everything in Python in the first place and it would have probably been much more readable.

> You could have done everything in Python in the first place and it would have probably been much more readable.

Python scripts that call a few third party programs are notoriously unreadable, full of subprocess call getouptut decode(utf8) bullshit. Python is alright as a language, but it is a very bad team player: it only really works when everything is written in Python, if you want to use things written in other languages it becomes icky really fast. Python projects that use parts written in other languages inevitably gravitate to being 100% Python. Another way to put it, is that Python is a cancer.

Re: The Beauty of Unix Pipelines

#212

Post by viraptor » Fri, Jun 05, 2020, 5:06 AM UTC

enriquto wrote: post #205

viraptor wrote: post #126

Earlier quoted context omitted.

You don't. If you use it once, meh. If you share it with anyone or preserve for the future, why would you want it to be cute? It's just a few lines in python, probably takes just as long to write because you don't have to play with what needs to be escaped and what doesn't. You can actually tell what's the intent of each line and it doesn't fail on paths starting with minuses or including spaces. Outside of one time…

This is not about "code golfing", the "sort|uniq -c|sort" combo is in the hall of fame of great code lines. The PATH thing in my script was an unnecessary distractor, consider this: file -b /bin/* /usr/bin/* |cut -d' ' -f-2|sort|uniq -c|sort -n There are no bizarre escapes nor anything. Besides, the "file" program is called only once. The python equivalent that you wrote may be better if you want to store it somewher…

> Moreover, you can run it everywhere

Issue 1: `zsh: argument list too long: file`

Issue 2: `file -b /usr/bin/apt` returns 3 lines, so the entries you count end up being:

    2 /usr/bin/apt (for
    1 Mach-O universal

Yes, it looks easier, but there are traps.

Re: The Beauty of Unix Pipelines

#213

Post by chubot » Fri, Jun 05, 2020, 5:07 AM UTC

antipaul wrote: post #181

A similar philosophy has made the "tidyverse" a much-loved extension of the statistical language R. Compare the following 2 equivalent snippets. Which one seems more understandable? iris_data %>% names() %>% tolower() %>% gsub(".", "_", ., fixed=TRUE) %>% paste0("(", ., ")") or: paste0("(", gsub(".", "_", tolower(names(iris_data)), fixed=TRUE), ")")

Yup, I'm working on Oil shell and I wrote an article about data frames, comparing tidyverse, Pandas, and raw Python:

What Is a Data Frame? (In Python, R, and SQL)

http://www.oilshell.org/blog/2018/11/30.html

The idea is essentially to combine tidyverse and shell -- i.e. put structured data in pipes. I use both shell and R for data cleaning.

My approach is different than say PowerShell because data still gets serialized; it's just more strictly defined and easily parseable. It's more like JSON than in-memory objects.

The left-to-right syntax is nicer and more composable IMO, and many functional languages are growing this feature (Scala, maybe Haskell?). Although I think Unix pipes serve a distinct use case.

Re: The Beauty of Unix Pipelines

#214

Post by asicsp » Fri, Jun 05, 2020, 5:08 AM UTC

young_unixer wrote: post #23

salmo wrote: post #20

Earlier quoted context omitted.

I think people can go through a few stages of their shell-foo. The first involves a lot of single commands and temporary files. The second uses pipes, but only tacks on commands with no refactoring. The third would recognize that all the grep and cut should just be awk, that you can redirect the cumulative output of a control statement, that subprocesses and coroutines are your friend. We should all aspire to this. T…

My most complex use of the shell is defining aliases in .bashrc and have never felt the need to go further. Do you recommend learning all of that for someone like me? If so, what resource do you recommend?

I have books on GNU grep/sed/awk [0] (currently free in spirit of quarantine learning) that teaches the command and features step by step using plenty of examples and exercises. There's a separate chapter for regular expressions too.

And I have a list of curated resources for `bash` and Linux [1]. Here's what I'd recommend:

* https://ryanstutorials.net/linuxtutorial/ is a good place to start for beginners

* https://mywiki.wooledge.org/BashGuide and the rest of this wonderful wonderful site is recommended to understand `bash` and its features and gotchas

* https://explainshell.com/ gives you a quick help on the various parts of the command, including documentation

* https://www.shellcheck.net/ is a must have tool if you are writing shell scripts

* https://linuxjourney.com/ is probably the most complete site to understand how things work in *nix environment

[0] https://learnbyexample.github.io/books/

[1] https://github.com/learnbyexample/scripting_course/blob/mast...

Re: The Beauty of Unix Pipelines

#215

Post by gitgud » Fri, Jun 05, 2020, 5:09 AM UTC

shrimp_emoji wrote: post #91

gorgoiler wrote: post #33

Earlier quoted context omitted.

Exactly. “ Linux, ew !” A big part of teaching computer science to children is breaking this obsession with approaching a computer from the top down — the old ICT ways, and the love of apps — and learning that it is a machine under your own control the understanding of which is entirely tractable from the bottom up. Unlike the natural sciences, computer science (like math) is entirely man made, to its advantage. No m…

My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin…

I don't get this. Linux is just the kernel, there's a variety of OS distributions which allow you to customise them infinitely to be what ever you want.

Re: The Beauty of Unix Pipelines

#216

Post by mehrdadn » Fri, Jun 05, 2020, 5:14 AM UTC

enriquto wrote: post #210

blacksqr wrote: post #120

Earlier quoted context omitted.

In the early 2k's I discovered that it was impossible to quote or escape spaces in filepaths in a makefile. I naively reported it to the Make maintainer as a bug. The maintainer responded by saying, basically, that the behavior was deeply baked in and was not going to change.

This is a nice feature of make. People who put spaces on filenames are justly punished by not being given the goodness of make.

I can't tell if you're being sarcastic but this can result in innocent users who don't even know this fact to suddenly overwrite or wipe their files when trying to build someone's project.

Re: The Beauty of Unix Pipelines

#217

Post by [deleted] » Fri, Jun 05, 2020, 5:30 AM UTC

yjftsjthsd-h wrote: post #185

shrimp_emoji wrote: post #91

Earlier quoted context omitted.

My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin…

- Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) It has its pain points, but the things that (ba)sh is good at, it's really good at and python, in my experience, doesn't compete. Dumb example: `tar czf shorttexts.tar.gz $(find . -type f | grep ^./...\.txt) && rsync shorttexts.tar.gz laptop:`. I could probably make a python script that did the equi…

[deleted]

Re: The Beauty of Unix Pipelines

#218

Post by catalogia » Fri, Jun 05, 2020, 5:31 AM UTC

shrimp_emoji wrote: post #91

gorgoiler wrote: post #33

Earlier quoted context omitted.

Exactly. “ Linux, ew !” A big part of teaching computer science to children is breaking this obsession with approaching a computer from the top down — the old ICT ways, and the love of apps — and learning that it is a machine under your own control the understanding of which is entirely tractable from the bottom up. Unlike the natural sciences, computer science (like math) is entirely man made, to its advantage. No m…

My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin…

> multiple users on one computer??

Last I checked this is something all the mainstream desktop operating systems support, so for you to call this out as a linux weirdness is baffling.

Re: The Beauty of Unix Pipelines

#219

Post by Quekid5 » Fri, Jun 05, 2020, 5:43 AM UTC

enriquto wrote: post #204

fomine3 wrote: post #175

Earlier quoted context omitted.

The fact that filename can contain anything except NULL/slash is really pain. I often write a shell script that treats LF as separator but I know it's not good.

filenames are variable names. Choosing good filenames is the first step in writing a nice shell script in a situation that you control.

No, they aren't -- they are data.

Re: The Beauty of Unix Pipelines

#220

Post by Quekid5 » Fri, Jun 05, 2020, 5:46 AM UTC

the_af wrote: post #57

cowmix wrote: post #4

I use pipelines as much as the next guy but every time I see post praise how awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on pipelines is pretty spot on too. http://web.mit.edu/~simsong/www/ugh.pdf

I think of the Unix Hater's Handbook as a kind of loving roast to Unix, that hackers of the time understood to be humorous (you know, how people complain about the tools they use every day, much like people would later complain endlessly about Windows) and which was widely misunderstood later to be a real scathing attack. It hasn't aged very well, either. "Even today, the X server turns fast computers into dumb termi…

> It hasn't aged very well, either. "Even today, the X server turns fast computers into dumb terminals" hasn't been true for at least a couple of decades...

You're not wrong, but that's only because people wrote extensions for direct access to the graphics hardware... which obviously don't work remotely, and so aren't really in the spirit of X. It's great that that was possible, but OTOH it probably delayed the invention of something like Wayland for a decade+.

It's been ages since I've used X for a remote application, but I sometimes wonder how many of them actually really still work to any reasonable degree. I remember some of them working when I last tried it, albeit with abysmal performance compared to Remote desktop, for example.