Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

371–380 of 388 posts

Re: The Beauty of Unix Pipelines

#371

Post by imglorp »

hhas01 wrote: post #344
imglorp wrote: post #45

Earlier quoted context omitted.

More Lego rules: (7) and common signalling facility (8) also nice if some files have magic properties like /dev/random or /proc or /dev/null (9) every program starts with 3 streams, stdin/stdout for work and stderr for out of band errors

I defy anyone to build their error-handling logic off the back of stderr…

Logic, not in this case, just messages.

The point of separate error streams echoes the main Unix philosophy of doing small things, on the happy path, that work quietly or break early. Your happy results can be chained on to the next thing and your errors go somewhere for autopsy if necessary. Eg,

    p1 | p2 | p3 > result.txt 2> errors.txt
will not contaminate your result file with any stderr gripes. You may need to arrange error streams from p1 and p2 using () or their own redirection. If it dies, it will die early.

BUT speaking of logic, the concept is close to Railway Oriented Programming, where you can avoid all the failure conditionals and just pass a monad out of a function, which lets you chain happy path neatly. Eg, nice talk here https://fsharpforfunandprofit.com/posts/recipe-part2/

Re: The Beauty of Unix Pipelines

#372

Post by [deleted] »

bityard wrote: post #71
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> (1) everything is text Not at all, you can pipe around all the binary you want. Until GNU tar added the 'z' option, the way to extract all files in a tarball was: `gunzip -c However, "text files" in Unix do have a very specific definition, if you want them to work with standard Unix text manipulation utilities like awk, sed, and diff: All lines of text end are terminated by a line feed, even the last one. I can't t…

[deleted]

Re: The Beauty of Unix Pipelines

#373

Post by hnlmorg »

buzzkillington wrote: post #321
hnlmorg wrote: post #268

Earlier quoted context omitted.

As long as you re-serialise that data when piping to coreutils (etc) you shouldn't have an issue. This is what my shell ( https://github.com/lmorg/murex ) does. It defaults to using JSON as a serialisation format (that's how arrays, maps, etc are stored, how spaces are escaped, etc) but it can re-serialise that data when piping into executables which aren't JSON aware. Other serialisation formats are also supported s…

Ascii records have worked better than json, xml, s-expressions (better as in provide the same bang for a lot less buck) in every case I've used them for. I have no idea why I am the only person I know who uses them regularly.

ASCII records have their own limitations though:

- They can be as easily edited by hand like other serialisation formats which use printable characters as their deliminators

- It's not clearly defined how you'd use them for non-tabulated data (such as JSON, XML, S-Expressions)

- There isn't any standard for escaping control characters

- They're harder to differentiate between unserialised binary formats

- They can't be used as a primitive like JSON is to Javascript and S-Expressions is to Lisp.

And if we're both honest, reducing the serialisation overhead doesn't gain you anything when working in the command line. It's a hard enough sell getting websites to support BSON and at least there, there is a tangible benefit of scale.

Not that I'm dismissing ASCII records, they would have been better than the whitespace mess we currently have in POSIX shells. However I don't agree ASCII records are better that current JSON nor S-Expressions.

Re: The Beauty of Unix Pipelines

#374

Post by russellbeattie »

mongol wrote: post #369
russellbeattie wrote: post #159

Earlier quoted context omitted.

Here's a JS script [1] I wrote a little while ago just for my own use that queries the CDC for the latest virus numbers, then calcs the average, formats the data and prints to the command line. You can pass in a number of days, otherwise it pulls the last 14 days. $ node query-cdc.js 7 It's nothing special, but I wouldn't want to try to do this with command line utilities. (And yes, it was a bit uglier, but I cleaned…

Thanks, appreciated! I guess using shebang #!/usr/bin/node does not work as # is not a valid comment?

I think Node has an exception for that built in, so you can do that or #!/usr/bin/env node and it'll work.

Re: The Beauty of Unix Pipelines

#375

Post by xkucf03 »

gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

Unix is seriously cool and I consider myself still young :-) But it could be better. See https://relational-pipes.globalcode.info/

Re: The Beauty of Unix Pipelines

#376

Post by xkucf03 »

tsimionescu wrote: post #257
gorgoiler wrote: post #47

Earlier quoted context omitted.

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools desig…

The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world. The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.

I agree that spaces, quotes, encodings, line-ends etc. are problem, but this is not problem of „the pipeline model“ – this is problem of ambiguously-structured data. See https://relational-pipes.globalcode.info/v_0/classic-example...

Re: The Beauty of Unix Pipelines

#379

Post by brokenmachine »

fomine3 wrote: post #175
jcranmer wrote: post #38

Earlier quoted context omitted.

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them…

The fact that filename can contain anything except NULL/slash is really pain. I often write a shell script that treats LF as separator but I know it's not good.

Why not use null as a separator?

I agree it's messy but not that hard. Just set IFS to null.

Re: The Beauty of Unix Pipelines

#380

Post by tandav »

ehsankia wrote: post #89
geophile wrote: post #5

I love pipelines. I don't know the elaborate sublanguages of find, awk, and others, to exploit them adequately. I also love Python, and would rather use Python than those sublanguages. I'm developing a shell based on these ideas: https://github.com/geophile/marcel .

+1 Piping is great if you memorize the (often very different) syntax of every individual tool and memorize their flags, but in reality unless it's a task you're doing weekly, you'll have to go digging through MAN pages and documentation every time. It's just not intuitive. Still to date if I don't use `tar` for a few months, I need to lookup the hodge podge of letters needed to make it work. Whenever possible, I just…

+1, but I use jupyter instead of IPython
Post reply on HN