Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

241–250 of 388 posts

Re: The Beauty of Unix Pipelines

#241
post #176

Earlier quoted context omitted.

Going with the same theme, C itself was an innovation meant to fill the space between Assembler (in this context, "A") and B[0]. [0] https://en.wikipedia.org/wiki/B_(programming_language)

That isn't really supported by the article you linked -- it describes C as an extension of B multiple times.

To be more precise: A->BCPL->B->C

Re: The Beauty of Unix Pipelines

#242
post #46

Earlier quoted context omitted.

That's a POSIX shell thing rather than a Unix pipeline thing. Some non-POSIX shells don't have this problem while still passing data long Unix pipes Source: I wrote a shell and it solves a great many of the space / quoting problems with POSIX shells.

Same, I've built toy shells [0] that used xml, s-expressions and ascii delimited text [1]. The last was closest to what unix pipes should be like. Of course it breaks ALL posix tools, but it felt like a shell that finally works as you'd expect it to work. [0] Really just input + exec + pipes. [1] https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

Here's one tool that it does not break. (-:

* http://jdebp.uk./Softwares/nosh/guide/commands/console-flat-...

Re: The Beauty of Unix Pipelines

#243

Earlier quoted context omitted.

God help you if your paths have spaces in them.

No need to invoke a deity to do that... echo 'Hi!' | tee "$(printf "foo\nbar\tqu ux.txt")" cat foo$'\n'bar$'\t'qu' 'ux.txt IIRC the latter only works in ksh93+, but the former should be fine pretty much in any shell.

... except if one uses \c by mistake. \c has unspecified behaviour, and is one of the things that varies widely from one implementation of printf to another.

* https://unix.stackexchange.com/a/558665/5132

Re: The Beauty of Unix Pipelines

#244
post #147
post #67

Earlier quoted context omitted.

My complaints with unix (as someone running linux on every device, starting to dip my toe into freebsd on a vps); apologies for lack of editing: > everything is text I'd often like to send something structured between processes without needing both sides to have to roll their own de/serialization of the domain types; in practice I end up using sockets + some thrown-together HTTP+JSON or TCP+JSON thing instead of pipe…

> As in Unix signals? tbh those seem ugly too; sigwinch, sighup, etc ought to be connected to stdin in some way; it'd be nice if there were a more general way to send arbitrary data to processes as an event Well, as the process can arbitrarily change which stdin it is connected to, as stdin is a file, you need some way to still issue directions to that process. However, for the general case, your terminal probably su…

Sure, I mean more like there'd be some interface ByteStream, and probably another interface TerminalInput extends ByteStream; TerminalInput would then contain the especially terminal-related signals (e.g. sigwinch). If a program doesn't declare that it wants a full TerminalInput for stdin, the sigwinches would be dropped. If it declares it wants one but the environment can't provide one (e.g. stdin is closed, stdin is a file, etc.) it would be an error from execve().

Other signals would be provided by other resources, ofc; e.g. sigint should probably be present in all programs.

---

In general, my ideal OS would have look something like [0], but with isolation between the objects (since they're now processes), provisions for state changes (so probably interfaces would look more like session types), and with a type system that supports algebraic data types.

[0]: https://www.tedinski.com/2018/02/20/an-oo-language-without-i...

Re: The Beauty of Unix Pipelines

#245
post #84
post #30

Earlier quoted context omitted.

> everything is text Everything is a byte stream. Usually that means text but sometimes it doesn't. Which means you can do fun stuff like: - copy file systems over a network: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm... - stream a file into gzip - backup or restore an SD card using `cat`

Also dd, which may be the disk destroyer but is also a great tool for binary file miracles. See one here: https://unix.stackexchange.com/questions/6852/best-way-to-re...

dd doesn’t example byte stream pipelines though ;)

But yes, that’s the typical file / block management tool. Just remember to set a block size otherwise you’d suffer from worse performance than using cat

Re: The Beauty of Unix Pipelines

#246

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them…

Or you just pipe through the program of your choice for which you know the syntax:

  echo "foo bar fnord" | cut --delimiter " " --output-delimiter ":" -f1-

  # foo:bar:fnord

Re: The Beauty of Unix Pipelines

#247

Earlier quoted context omitted.

You don't. If you use it once, meh. If you share it with anyone or preserve for the future, why would you want it to be cute? It's just a few lines in python, probably takes just as long to write because you don't have to play with what needs to be escaped and what doesn't. You can actually tell what's the intent of each line and it doesn't fail on paths starting with minuses or including spaces. Outside of one time…

This is not about "code golfing", the "sort|uniq -c|sort" combo is in the hall of fame of great code lines. The PATH thing in my script was an unnecessary distractor, consider this: file -b /bin/* /usr/bin/* |cut -d' ' -f-2|sort|uniq -c|sort -n There are no bizarre escapes nor anything. Besides, the "file" program is called only once. The python equivalent that you wrote may be better if you want to store it somewher…

find-xargs combo could have been better. It also prevents the case where you cross arg max

Re: The Beauty of Unix Pipelines

#249

Earlier quoted context omitted.

> Pipes are wonderful! It's wonderful only if compared to worse things, pretending that PowerShell is not a thing, and that Python doesn't exist. UNIX pipes are a stringly-typed legacy that we've inherited from the 1970s. The technical constraints of its past have been internalised by its proponents, and lauded as benefits . To put it most succinctly, the "byte stream" nature of UNIX pipes means that any command that…

> Where each one of those "parse" and "serialise" steps is unique and special, inflexible, and poorly documented. This can also be a security concern. According to research, a great number of defects with security implications occur at the input handling layers: http://langsec.org

Famously, "avoid parsing" was one of the qmail security principles (q.v.).

Re: The Beauty of Unix Pipelines

#250
post #150

Earlier quoted context omitted.

I am firmly in the "ugh" camp. I strongly suspect that the fawning that occurs over pipelines is because of sentimentality more than practicality. Extracting information from text using a regular expression is fragile. Writing fragile code brings me absolutely no joy as a programmer - unless I am trying to flex my regex skills. If you really look at how pipelines are typically used is: lines are analogous to objects…

If you're having to extract your data using a regex, then the data probably isn't well-formed enough for a shell pipeline. It's doable, but a bad idea. Regex should not be the first hammer you reach for, because it's a scalpel. I recently wanted cpu cores + 1. That could be a single regex. But this is more maintainable, and readable: echo '1 + '"$(grep 'cpu cores' /proc/cpuinfo | tail -n1 | awk -F ':' '{print $2}')"…

This is where you need to use awk.

    awk -F ':' '/cpu cores/ {a = $2 + 1} END {print a}' /proc/cpuinfo
Post reply on HN