Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

321–330 of 388 posts

Re: The Beauty of Unix Pipelines

#321

Post by buzzkillington »

hnlmorg wrote: post #268
buzzkillington wrote: post #172

Earlier quoted context omitted.

Same, I've built toy shells [0] that used xml, s-expressions and ascii delimited text [1]. The last was closest to what unix pipes should be like. Of course it breaks ALL posix tools, but it felt like a shell that finally works as you'd expect it to work. [0] Really just input + exec + pipes. [1] https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

As long as you re-serialise that data when piping to coreutils (etc) you shouldn't have an issue. This is what my shell ( https://github.com/lmorg/murex ) does. It defaults to using JSON as a serialisation format (that's how arrays, maps, etc are stored, how spaces are escaped, etc) but it can re-serialise that data when piping into executables which aren't JSON aware. Other serialisation formats are also supported s…

Ascii records have worked better than json, xml, s-expressions (better as in provide the same bang for a lot less buck) in every case I've used them for.

I have no idea why I am the only person I know who uses them regularly.

Re: The Beauty of Unix Pipelines

#322

Post by Icathian »

asdff wrote: post #148
kccqzy wrote: post #99

Earlier quoted context omitted.

It replaces the executable in the current process with a different executable. It's kind of like spawning a new process with an executable, except it's the same PID, and file descriptors without CLOEXEC remain open in their original state.

whats a simple example of using this?

Bash is probably the best example of this style of working. You have a shell process that forks and spawns child processes from itself that are turned into whatever program you've called from your terminal.

Re: The Beauty of Unix Pipelines

#323

Post by xnyan »

dvirsky wrote: post #135
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> Unix is seriously uncool with young people at the moment. Those damn kids with their loud rockn'roll music and their Windows machines. Back in my day we had he vocal stylings of Dean Martin and the verbal stylings of Linus Torvalds let me tell ya. Seriously though, I'm actually seeing younger engineers really taking the time to learn how to do shell magic, using vim, etc. It's like the generation of programmers who…

I grew up all-GUI windows kid. I actually had a revulsion to the shell, mostly because it seemed unfriendly and scary. In my early 20s I tried to run Plex on a Windows 7 box and it was miserable. I forced myself to lean by switching to a headless arch linux box.

Giving up the idea that CLI = pain (i.e figuring out how to to navigate a file system, ssh keys, etc) for sure was a learning curve, but now I can't imagine using computers without it.

Re: The Beauty of Unix Pipelines

#324

Post by mbreese »

tsimionescu wrote: post #257
gorgoiler wrote: post #47

Earlier quoted context omitted.

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools desig…

The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world. The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.

Pipelines are mainly for short-lived, one-off quick scripts. But they are also really useful for when you control the input data. For example, if you need a summary report based on the results of a sql query.

If you control the inputs and you need to support quotes, spaces, non-white space delimiters, etc... in shell script, then that’s on you.

If you don’t control the inputs, then shell scripts are generally a poor match. For example, if you need summary reports from a client, but they sometimes provide the table in xlxs or csv format — shell might not be a good idea.

Might be controversial, but I think you can tell who works with shell pipes the most by looking at who uses CSV vs tab-delimited text files. Tabs can still be a pain if you have spaces in data. But if you mix shell scripts with CSV, you’re just asking for trouble.

Re: The Beauty of Unix Pipelines

#325

Post by the_af »

Quekid5 wrote: post #220
the_af wrote: post #57

Earlier quoted context omitted.

I think of the Unix Hater's Handbook as a kind of loving roast to Unix, that hackers of the time understood to be humorous (you know, how people complain about the tools they use every day, much like people would later complain endlessly about Windows) and which was widely misunderstood later to be a real scathing attack. It hasn't aged very well, either. "Even today, the X server turns fast computers into dumb termi…

> It hasn't aged very well, either. "Even today, the X server turns fast computers into dumb terminals" hasn't been true for at least a couple of decades... You're not wrong, but that's only because people wrote extensions for direct access to the graphics hardware... which obviously don't work remotely, and so aren't really in the spirit of X. It's great that that was possible, but OTOH it probably delayed the inven…

Fair enough! I wasn't really thinking of the remote use case which was indeed X's main use case when it was conceived (and which is not very relevant today for the majority of users).

In any case, this was just an example. The Handbook is peppered with complaints which haven't been relevant for ages. It was written before Linux got user-friendly UIs and was widespread to almost every appliance on Earth. It was written before Linux could run AAA games. It was written before Docker. It was written before so many people knew how to program simple scripts. It was written before Windows and Apple embraced Unix.

If some of these people are still living, I wonder what they think of the (tech) world today. Maybe they are still bitter, or maybe they understand how much they missed the mark ;)

Re: The Beauty of Unix Pipelines

#326

Post by praveen9920 »

I recently wrote a golang code for fetching rss feed and displaying gist based on requirement.

Looking at this code, I am tempted to reimplement this using pipes but saner mind took over and said "don't fix something that is not broken"

I probably would be still do it and get some benchmark numbers to compare both.

Re: The Beauty of Unix Pipelines

#327

Post by bloopernova »

jlg23 wrote: post #246
jcranmer wrote: post #38

Earlier quoted context omitted.

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them…

Or you just pipe through the program of your choice for which you know the syntax: echo "foo bar fnord" | cut --delimiter " " --output-delimiter ":" -f1- # foo:bar:fnord

I love learning new things about commands I've used for years.

    --output-delimiter ":"
Very cool!

However it does require the GNU version of cut, not the Mac OS X supplied BSD version. In zsh at least, you can do:

    alias cut='/usr/local/bin/gcut'

Re: The Beauty of Unix Pipelines

#328

Post by anthk »

pjmlp wrote: post #230
anthk wrote: post #96

Earlier quoted context omitted.

- Python as a shell would be the worst crapware ever. Whitespace syntax, no proper file autocompletion, no good pipes, no nothing. Even Tclsh with readline would be better than a Python shell. - By mixing C and C++ you look like a clueless youngster. - /usr/local is to handle non-base/non-packages stuff so you don't trash your system. Under OpenBSD, /usr/local is for packages, everything else should be under /opt or…

The ISO C and C++ papers are full of C/C++ references, are the ISO masters clueless youngsters? Or for that matter the likes from Apple, Google and Microsoft?

EVen C99 has nothing to do with bastard childs such as C++.

The last is convoluted crap, full of redundant functions.

Further C standards do exist, but C99 and heck, even ANSI C can be good enough for daily uses.

Perl, Ruby and PHP look close to each other, being the first the source of inspiration.

But don't try to write PHP/Ruby code as if it was something close to Perl. The same with C++.

Re: The Beauty of Unix Pipelines

#329

Post by nerdponx »

tsimionescu wrote: post #257
gorgoiler wrote: post #47

Earlier quoted context omitted.

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools desig…

The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world. The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.

Kind of / not really.

Also Zsh solves 99% of my "pipelines are annoying" and "shell scripts are annoying" problems.

Even on systems where I can't set my default shell to Zsh, I either use it anyway inside Tmux, or I just use it for scripting. I suppose I use Zsh the way most people use Perl.

Re: The Beauty of Unix Pipelines

#330

Post by nerdponx »

enriquto wrote: post #258
jcranmer wrote: post #69

Earlier quoted context omitted.

There's a certain irony in responding to criticism of that which you're extolling by saying not to use it. And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.

> because spaces are a reasonable thing to put in a filename unless you're on a Unix system. Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.

I disagree with this immensely.

Also, some languages do allow whitespace in variable names like R and SQL, so long as the variable names are quoted or escaped properly.

Post reply on HN