Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

271–280 of 388 posts

Re: The Beauty of Unix Pipelines

#271

I wish there were gui pipes. Pipes are wonderful but they’re neither interactive nor continuous. Pipes that loop, outputting a declarative gui text format, and listen for events from the gui, would be marvellous. I can’t think how to do that without sockets and a bash loop. And that seems to create the kind of complexity that pipes manage to avoid.

[deleted]

Re: The Beauty of Unix Pipelines

#272
post #22

Earlier quoted context omitted.

Every time I see that sort of stuff I end up | perl, still after 20 odd years. Sometimes i fine perl is not installed and I die a little.

In what unix system perl is not installed? It is in the default install on current versions of macos (where it comprises the majority of interpreted programs), openbsd, freebsd, ubuntu and fedora linux.

Usually containers these days.

Re: The Beauty of Unix Pipelines

#273

Earlier quoted context omitted.

> (2) everything (ish) is a file I'm not all that knowledgable about Unix history, but one thing that has always puzzled me was that for whatever reason network connections (generally) aren't files. While I can do: cat to read from a serial device, I've always wondered why I can't do something like: bind /tmp/mysocket 12.34.56.78 80 cat It is weird that so many things in Unix are somehow twisted into files (/dev/fb0?…

elsewhere in this thread somebody mentions socat, but you can do it entirely within bash. from https://www.linuxjournal.com/content/more-using-bashs-built-... : exec 3 /dev/tcp/www.google.com/80 echo -e "GET / HTTP/1.1\r\nhost: http://www.google.com\r\nConnection: close\r\n\r\n" >&3 cat https://news.ycombinator.com/item?id=23422423 has a good point that "everything is a file" is maybe less useful than "everything is…

Daniel J. Bernstein's little known http@ tool is just a shell script wrapped around tcpclient, and is fairly similar, except that it does not use the Bashisms of course:

    printf 'GET /%s HTTP/1.0\nHost: %s:%s\n\n' "${2-}" "${1-0}" "${3-80}" |
    /usr/local/bin/tcpclient -RHl0 -- "${1-0}" "${3-80}" sh -c '
    /usr/local/bin/addcr >&7
    exec /usr/local/bin/delcr 

Re: The Beauty of Unix Pipelines

#274

If you eval this simple pipeline file -b `echo $PATH:|sed 's/:/\/* /g'`|cut -d\ -f-2|sort|uniq -c|sort -n it prints a histogram of the types of all the programs on your path (e.g., whether they are shell, python, perl scripts or executable binaries). How can you ever write such a cute thing in e.g., python or, god forbid, java?

- This fails on badly formed PATH, or malicious PATH

- When PATH contains entries that no longer exist / are not mounted, substitution with "sed" gives a wildcard that is not expanded, and eventually makes "file" report an error, which is not filtered out

- If PATH contains entries that have spaces, the expansion is incorrect

Re: The Beauty of Unix Pipelines

#275
post #250
post #150

Earlier quoted context omitted.

If you're having to extract your data using a regex, then the data probably isn't well-formed enough for a shell pipeline. It's doable, but a bad idea. Regex should not be the first hammer you reach for, because it's a scalpel. I recently wanted cpu cores + 1. That could be a single regex. But this is more maintainable, and readable: echo '1 + '"$(grep 'cpu cores' /proc/cpuinfo | tail -n1 | awk -F ':' '{print $2}')"…

This is where you need to use awk. awk -F ':' '/cpu cores/ {a = $2 + 1} END {print a}' /proc/cpuinfo

> This is where you need to use awk.

I think need is a strong statement in this context.

  echo $(( 1 + $(grep 'cpu cores' /proc/cpuinfo | head -1 | awk -F ':' '{print $2}') ))

Choosing shell vs. [insert modern] programming is a matter of trade-off of taste, time, $$$, collective consensus (in a team setting) and so forth.

[edit]: Forgot code syntax on HN.

Re: The Beauty of Unix Pipelines

#276

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> Pipes are wonderful! It's wonderful only if compared to worse things, pretending that PowerShell is not a thing, and that Python doesn't exist. UNIX pipes are a stringly-typed legacy that we've inherited from the 1970s. The technical constraints of its past have been internalised by its proponents, and lauded as benefits . To put it most succinctly, the "byte stream" nature of UNIX pipes means that any command that…

   The reality is that physically, it actually works like this:

      (process1,serialize) | (parse,process2,serialize) | ...
But as soon as you involve the network or persistent storage, you need to do all that anyway. And one of the beauties is that the tools are agnostic to where the data comes from, or goes.

Re: The Beauty of Unix Pipelines

#277
post #274

If you eval this simple pipeline file -b `echo $PATH:|sed 's/:/\/* /g'`|cut -d\ -f-2|sort|uniq -c|sort -n it prints a histogram of the types of all the programs on your path (e.g., whether they are shell, python, perl scripts or executable binaries). How can you ever write such a cute thing in e.g., python or, god forbid, java?

- This fails on badly formed PATH, or malicious PATH - When PATH contains entries that no longer exist / are not mounted, substitution with "sed" gives a wildcard that is not expanded, and eventually makes "file" report an error, which is not filtered out - If PATH contains entries that have spaces, the expansion is incorrect

More realistically, it also fails if you have directories on your path that are symlinks to other directories in the path. In that case their programs are doubly-counted.

Anyways, if your PATH is malicious then you have worse problems than this silly script :)

Re: The Beauty of Unix Pipelines

#278
post #267

Earlier quoted context omitted.

My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin…

> - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) You know that no one uses Bash these days right? Fish shell, Zsh are the much newer and friendlier shells these days. All my shells run on Fish these days. I have to tell you that my shell ergonomics has improved a lot after I started using fish shell. But to your point, _why_ is Bash awful?

> You know that no one uses Bash these days right?

I strongly disagree although, just like you, I have no evidence to support my claim

Re: The Beauty of Unix Pipelines

#279
post #83

Earlier quoted context omitted.

A little bit of grep / awk goes very, very far.

grep, awk, sed, cut, sort, uniq and join are the Swiss Army knife of working with tabulated data on the command line.

No, they are the opposites of an all-in-one tool.

For a Swiss Army Knife, look at the likes of Miller.

* https://johnkerl.org/miller/

Re: The Beauty of Unix Pipelines

#280
post #232

Earlier quoted context omitted.

Can you give a working example? This way you can watch a UNIX user "struggle/fail" as they "attempt" to debug a script using set -x, set -e, $?, LINENO, utilties like ktrace, strace, recordio, etc. and your argument will be taken seriously.

Some people also insist to keep debuging with printf(), whereas the rest of the world is happily using InteliJ and friends.

I just use gdb and ald. I heard IntelliJ is a memory hog and requires a powerful computer.
Post reply on HN