Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

71–80 of 388 posts

Re: The Beauty of Unix Pipelines

#71

Post by bityard »

gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> (1) everything is text

Not at all, you can pipe around all the binary you want. Until GNU tar added the 'z' option, the way to extract all files in a tarball was:

`gunzip -c However, "text files" in Unix do have a very specific definition, if you want them to work with standard Unix text manipulation utilities like awk, sed, and diff:

All lines of text end are terminated by a line feed, even the last one.

I can't tell you how many "text files" I run across these days that don't have a trailing newline. It's as bad as mixing tabs and spaces, maybe worse.

Re: The Beauty of Unix Pipelines

#72

Post by msla »

AnimalMuppet wrote: post #43
CalmStorm wrote: post #25

Unix pipelines are indeed beautiful, especially when you consider its similarity to Haskell's monadic I/O: http://okmij.org/ftp/Computation/monadic-shell.html Unix pipelines actually helped me make sense of Haskell's monad.

Could you be more specific? I don't get it.

It helps if you avoid the syntactic sugar of do-notation:

    main = getArgs >>= processData >>= displayData
main is in the IO monad. The >>= function takes a value wrapped in a monad and a function which accepts a value and returns a value wrapped in the same type of monad, and returns the same monad-wrapped value the function did. It can be used as an infix operator because Haskell allows that if you specify precedence, in which case its left side is the first argument (the value-in-a-monad) and its right side is the second argument (the function).

The (imaginary) function getArgs takes no arguments and returns a list of command-line arguments wrapped in an IO monad. The first >>= takes that list and feeds it into the function on its right, processData, which accepts it and returns some other value in an IO monad, which the third >>= accepts and feeds into displayData, which accepts it and returns nothing (or, technically, IO (), the unit (empty) value wrapped in an IO monad) which is main's return value.

See? Everything is still function application, but the mental model becomes feeding data through a pipeline of functions, each of which operates on the data and passes it along.

Re: The Beauty of Unix Pipelines

#73

Post by junke »

ken wrote: post #39
cowmix wrote: post #4

I use pipelines as much as the next guy but every time I see post praise how awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on pipelines is pretty spot on too. http://web.mit.edu/~simsong/www/ugh.pdf

> The Macintosh model, on the other hand, is the exact opposite. The system doesn’t deal with character streams. Data files are extremely high level, usually assuming that they are specific to an application. When was the last time you piped the output of one program to another on a Mac? And yet, at least it's possible to have more than one Macintosh application use the same data. Half the world has migrated to web a…

Command-line tools where you glue byte streams are, in spirit, very much like web scraping. Sure, tools can be good or bad, but by design you are going to have ad-hoc input/output formats, and for some tools this is interleaved with presentation/visual stuff (colors, alignment, headers, whitespaces, etc.).

In a way web apps are then more alike standard unix stuff, where you parse whatever output you get, hoping that it has enough usable structure to do an acceptable job.

The most reusable web apps are those that offer an API, with JSON/XML data formats where you can easily automate your work, and connect them together.

Re: The Beauty of Unix Pipelines

#74

Post by adamnemecek »

Unix pipes are the a 1970's construct, the same way bell bottom pants are. It's a construct that doesn't take into account the problems and scale of today's computing. Unicode? Hope your pipes process it fine. Video buffers? High perf? Fuggetaboutit. Piping the output of ls to idk what? Nice, I'll put it on the fridge.

Re: The Beauty of Unix Pipelines

#75

Post by ssivark »

gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

Serialized bytestreams do compose better than graphical applications. But that is setting a very low bar. For example, allowing passing around dicts/maps/json (and possibly other data structures) would already be a massive improvement. You know what might be even better — passing around objects you could interact with by passing messages (gasp! cue, Alan Kay).

While posix and streams are nice (if you squint, files look like a poor man’s version of objects imposed on top of bytestreams), it’s about time we moved on to better things, and not lose sight of the bigger picture.

Re: The Beauty of Unix Pipelines

#76

Post by matheusmoreira »

gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

1. All data is bytes

2. Everything is a file descriptor

3. File descriptors are things that support standard I/O system calls such as read and write

Re: The Beauty of Unix Pipelines

#77

Post by coliveira »

jcranmer wrote: post #69
gorgoiler wrote: post #47

Earlier quoted context omitted.

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools desig…

There's a certain irony in responding to criticism of that which you're extolling by saying not to use it. And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.

The shell works with spaces, you just need to be careful to quote every file name. The advantage of using file names without spaces is that you can avoid the quotes.

Re: The Beauty of Unix Pipelines

#78

Post by [deleted] »

6gvONxR4sf7o wrote: post #40
jcranmer wrote: post #38

Earlier quoted context omitted.

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them…

God help you if your paths have spaces in them.

[deleted]

Re: The Beauty of Unix Pipelines

#79

Post by jtbayly »

gorgoiler wrote: post #33
ianai wrote: post #27

Earlier quoted context omitted.

How’s it uncool? Just not “ mobile”?

Exactly. “ Linux, ew !” A big part of teaching computer science to children is breaking this obsession with approaching a computer from the top down — the old ICT ways, and the love of apps — and learning that it is a machine under your own control the understanding of which is entirely tractable from the bottom up. Unlike the natural sciences, computer science (like math) is entirely man made, to its advantage. No m…

Until stuff goes wrong with the hardware...
Post reply on HN