Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

261–270 of 388 posts

Re: The Beauty of Unix Pipelines

#261

Post by JdeBP »

quickthrower2 wrote: post #87
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

Can you give me any clue as to what execve does? I looked at the man page but none the wiser. Sounds like magic from what I read there. I'm from a Windows backgroud and not used to pipes.

Well you have a Windows background, so you know BASIC, right? And you know that BASIC has a CHAIN statement, right? (-:

Yes, I know. You've probably never touched either one. But the point is that this is simply chain loading. It's a concept not limited to Unix. execve() does it at the level of processes, where one program can chain to another one, both running in a single process. But it is most definitely not magic, nor something entirely alien to the world of Windows.

* https://en.wikibooks.org/wiki/QBasic/Appendix#CHAIN

If you are not used to pipes on Windows, you haven't pushed Windows nearly hard enough. The complaint about Microsoft DOS was that it didn't have pipes. But Windows NT has had proper pipes all along since the early 1990s, as OS/2 did before it since 1987. Microsoft's command interpreter is capable of using them, and all of the received wisdom that people knew about pipes and Microsoft's command interpreters on DOS rather famously (it being much discussed at the time) went away with OS/2 and Windows NT.

And there are umpteen ways of improving on that, from JP Software's Take Command to various flavours of Unix-alike tools. And you should see some of the things that people do with FOR /F .

Unix was the first with the garden hosepipe metaphor, but it has been some 50 years since then. It hasn't been limited to Unix for over 30 of them. Your operating system has them, and it is very much worthwhile investigating them.

Re: The Beauty of Unix Pipelines

#262

Post by enriquto »

outadoc wrote: post #259
enriquto wrote: post #258

Earlier quoted context omitted.

> because spaces are a reasonable thing to put in a filename unless you're on a Unix system. Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.

You should take a stroll in the real world sometime, where spaces and Unicode exists :)

Yeah, any unicode character is OK on a filename, except maybe space and slash.

Re: The Beauty of Unix Pipelines

#263

Post by lsofzz »

matheusmoreira wrote: post #183
jiggawatts wrote: post #141

Earlier quoted context omitted.

> Pipes are wonderful! It's wonderful only if compared to worse things, pretending that PowerShell is not a thing, and that Python doesn't exist. UNIX pipes are a stringly-typed legacy that we've inherited from the 1970s. The technical constraints of its past have been internalised by its proponents, and lauded as benefits . To put it most succinctly, the "byte stream" nature of UNIX pipes means that any command that…

> Where each one of those "parse" and "serialise" steps is unique and special, inflexible, and poorly documented. This can also be a security concern. According to research, a great number of defects with security implications occur at the input handling layers: http://langsec.org

> http://langsec.org

The maintainer apparently has no transport security concerns. It takes literally two minutes to set up TLS these days

Re: The Beauty of Unix Pipelines

#264

Post by teddyh »

tsimionescu wrote: post #255
coliveira wrote: post #77

Earlier quoted context omitted.

The shell works with spaces, you just need to be careful to quote every file name. The advantage of using file names without spaces is that you can avoid the quotes.

The shell (bash, anyway) has a weird mix of support and lack of support for spaces/lists/etc. For example, there is no way to store the `a b "c d"` in a regular variable in a way where you can then call something similar to `ls $VAR` and get the equivalent of `ls a b "c d"`. You can either get the behavior of `ls a b c d` or `ls "a b c d"`, but if you need `ls a b "c d"` you must go for an array variable with new syn…

That has nothing to do with spaces, but more to do with the rather lackluster support for arrays in most Bourne-style shells.

Re: The Beauty of Unix Pipelines

#265

Post by JdeBP »

remexre wrote: post #238
blibble wrote: post #145

Earlier quoted context omitted.

> iow, things get trickier once I need more than one input one one output. :) you can pass as many FDs to a child as you want, 0/1/2 is just a convention

At that point you're writing something for /usr/libexec, not /usr/bin though. (i.e., at that point it becomes sufficiently inconvenient to use from the shell that no user-facing program would do so.)

That's pretty much false on every point.

* https://unix.stackexchange.com/a/331104/5132

Re: The Beauty of Unix Pipelines

#266

Post by JdeBP »

jandrese wrote: post #166
ryandrake wrote: post #104

Earlier quoted context omitted.

> (2) everything (ish) is a file I'm not all that knowledgable about Unix history, but one thing that has always puzzled me was that for whatever reason network connections (generally) aren't files. While I can do: cat to read from a serial device, I've always wondered why I can't do something like: bind /tmp/mysocket 12.34.56.78 80 cat It is weird that so many things in Unix are somehow twisted into files (/dev/fb0?…

Berkeley sockets were kind of bolted on as an afterthought. If the AT&T guys developed them they would probably look a lot more like that.

There's no need for a subjunctive. Go and look at AT&T STREAMS and see what actually did result.

Re: The Beauty of Unix Pipelines

#267

Post by lsofzz »

shrimp_emoji wrote: post #91
gorgoiler wrote: post #33

Earlier quoted context omitted.

Exactly. “ Linux, ew !” A big part of teaching computer science to children is breaking this obsession with approaching a computer from the top down — the old ICT ways, and the love of apps — and learning that it is a machine under your own control the understanding of which is entirely tractable from the bottom up. Unlike the natural sciences, computer science (like math) is entirely man made, to its advantage. No m…

My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin…

> - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion)

You know that no one uses Bash these days right?

Fish shell, Zsh are the much newer and friendlier shells these days. All my shells run on Fish these days.

I have to tell you that my shell ergonomics has improved a lot after I started using fish shell.

But to your point, _why_ is Bash awful?

Re: The Beauty of Unix Pipelines

#268

Post by hnlmorg »

buzzkillington wrote: post #172
hnlmorg wrote: post #46

Earlier quoted context omitted.

That's a POSIX shell thing rather than a Unix pipeline thing. Some non-POSIX shells don't have this problem while still passing data long Unix pipes Source: I wrote a shell and it solves a great many of the space / quoting problems with POSIX shells.

Same, I've built toy shells [0] that used xml, s-expressions and ascii delimited text [1]. The last was closest to what unix pipes should be like. Of course it breaks ALL posix tools, but it felt like a shell that finally works as you'd expect it to work. [0] Really just input + exec + pipes. [1] https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

As long as you re-serialise that data when piping to coreutils (etc) you shouldn't have an issue.

This is what my shell (https://github.com/lmorg/murex) does. It defaults to using JSON as a serialisation format (that's how arrays, maps, etc are stored, how spaces are escaped, etc) but it can re-serialise that data when piping into executables which aren't JSON aware.

Other serialisation formats are also supported such as YAML, CSV and S-Expressions too. I had also considered adding support for ASCII records but it's not a use case I personally run into (unlike JSON, YAML, CSV, etc) so haven't written a marshaller to support it yet.

Re: The Beauty of Unix Pipelines

#269

Post by tarkin2 »

I wish there were gui pipes. Pipes are wonderful but they’re neither interactive nor continuous.

Pipes that loop, outputting a declarative gui text format, and listen for events from the gui, would be marvellous.

I can’t think how to do that without sockets and a bash loop. And that seems to create the kind of complexity that pipes manage to avoid.

Re: The Beauty of Unix Pipelines

#270

Post by junke »

junke wrote: post #49
Aaronstotle wrote: post #10

Earlier quoted context omitted.

I wouldn't be so quick to say it's an alt-right joke. Plenty of my friends will describe something/someone as a Chad and it has nothing to do with incel/alt-right cultures. Are those kind of phrases thrown out in those circles? Yes, but that's more of the general meme/internet lingo as opposed to subscribing to an ideology.

This is brogrammer speak (e.g. http://new_words.enacademic.com/662/brogrammer ).

(can't believe this was downvoted)
Post reply on HN