Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

121–130 of 388 posts

Re: The Beauty of Unix Pipelines

#121
post #113
post #75

Earlier quoted context omitted.

Serialized bytestreams do compose better than graphical applications. But that is setting a very low bar. For example, allowing passing around dicts/maps/json (and possibly other data structures) would already be a massive improvement. You know what might be even better — passing around objects you could interact with by passing messages (gasp! cue, Alan Kay). While posix and streams are nice (if you squint, files lo…

Yeah, but that makes every single file an object. Then you need plugins to deal with plain text objects, plugins to deal with every single type of object and their versions. Objects aren't human readable and if they get corrupted the object's format is very difficult to recover, you need an intimate knowledge of both the specific format used (of which there might be thousands of variations). Plaintext, however, if th…

But that seems like just ignoring the problem, like an ostrich shoving one’s head into the sand!

> Have you ever examined a protobuf file without knowing what the protobuf structure is?

Are you referring to an ASCII serialization or a binary serialization?

In a typical scenario, your application needed structured data (hence protobuf), and then you serialized your protobuf into an ASCII bytestream... so how are you ever worse off with a protobuf compared to a text file, provided you make a good choice of serialization scheme? Clearly text files cannot carry enough structure, so we have to build more layers of scaffolding on top?

Ultimately files are serialized on to disk as bits, and “ASCII” and “posix” are just “plugins” to be used when reading bitstreams (serialized file objects). Plaintext is not readable if the bits get corrupted or byte boundaries/endianness gets shifted!

To have the same amount of of robustness with higher-order structures I imagine all one needs is a well-defined de/serialization protocol which would allow you to load new objects into memory? Am I missing something?

Re: The Beauty of Unix Pipelines

#123
I think it will be on topic if I let myself take this occasion to once again plug in a short public service announcement of an open-source tool I built, that helps interactively build Unix/Linux pipelines, dubbed "The Ultimate Plumber":

https://github.com/akavel/up/

I've also recently seen it being described in shorter words as a "sticky REPL for shell". Hope you like it, and it makes your life easier!

Re: The Beauty of Unix Pipelines

#124

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them…

People struggle so much with this, but I don't see what the point is at all. The fundamental problem that playing with delimiters solves, is passing arbitrary strings through as single tokens.

Well, that's easy. Don't escape or quote the strings; encode them. Turn them into opaque tokens, and then do Unix things to the opaque tokens, before finally decoding them back to being strings.

There's a reason od(1) is in coreutils. It's a Unix fundamental when working with arbitrary data. Hex and base64 are your friends (and Unix tools are happy to deal with both.)

Re: The Beauty of Unix Pipelines

#125
post #65
post #57

Earlier quoted context omitted.

I think of the Unix Hater's Handbook as a kind of loving roast to Unix, that hackers of the time understood to be humorous (you know, how people complain about the tools they use every day, much like people would later complain endlessly about Windows) and which was widely misunderstood later to be a real scathing attack. It hasn't aged very well, either. "Even today, the X server turns fast computers into dumb termi…

No, I think it's mostly just as angry and bitter as it sounds, coming from people who backed the wrong horse (ITS, Lisp Machines, the various things Xerox was utterly determined to kill off... ) and got extremely pissy about Unix being the last OS concept standing outside of, like, VMS or IBM mainframe crap, neither of which they'd see as improvements.

Just because Unix was the last man standing, doesn't mean it's good. Bad products win in the marketplace all the time and for all kinds of reasons. In Unix's case I'd argue the damage is particularly severe because hackers have elevated Unix to a religion and insisted that all its flaws are actually virtues.

Re: The Beauty of Unix Pipelines

#126

If you eval this simple pipeline file -b `echo $PATH:|sed 's/:/\/* /g'`|cut -d\ -f-2|sort|uniq -c|sort -n it prints a histogram of the types of all the programs on your path (e.g., whether they are shell, python, perl scripts or executable binaries). How can you ever write such a cute thing in e.g., python or, god forbid, java?

You don't. If you use it once, meh. If you share it with anyone or preserve for the future, why would you want it to be cute?

It's just a few lines in python, probably takes just as long to write because you don't have to play with what needs to be escaped and what doesn't. You can actually tell what's the intent of each line and it doesn't fail on paths starting with minuses or including spaces. Outside of one time use or a code golf challenge, it's not cute.

https://pastebin.com/FnBCgHUm

Re: The Beauty of Unix Pipelines

#127
post #96

Earlier quoted context omitted.

My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin…

- Python as a shell would be the worst crapware ever. Whitespace syntax, no proper file autocompletion, no good pipes, no nothing. Even Tclsh with readline would be better than a Python shell. - By mixing C and C++ you look like a clueless youngster. - /usr/local is to handle non-base/non-packages stuff so you don't trash your system. Under OpenBSD, /usr/local is for packages, everything else should be under /opt or…

> Even Tclsh with readline would be better than a Python shell

In fact, I think Tcl would make an excellent sh replacement.

Re: The Beauty of Unix Pipelines

#128
I am sorry for playing the devil's advocate. I also think that pipes are extremely useful and a very strong paradigm, and I use them daily in my work. Also it is not an accident that it is fundamental and integral part of powershell too.

But is this really HN top page worthy? I have seen this horse beaten to death for decades now. These kind of articles have been around since the very beginning of the internet.

Am I missing something newsworthy which makes this article different from the hundreds of thousands of similar articles?

Re: The Beauty of Unix Pipelines

#129

Earlier quoted context omitted.

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them…

God help you if your paths have spaces in them.

No need to invoke a deity to do that...

  echo 'Hi!' | tee "$(printf "foo\nbar\tqu ux.txt")"
  cat foo$'\n'bar$'\t'qu' 'ux.txt
IIRC the latter only works in ksh93+, but the former should be fine pretty much in any shell.

Re: The Beauty of Unix Pipelines

#130
post #96

Earlier quoted context omitted.

My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin…

- Python as a shell would be the worst crapware ever. Whitespace syntax, no proper file autocompletion, no good pipes, no nothing. Even Tclsh with readline would be better than a Python shell. - By mixing C and C++ you look like a clueless youngster. - /usr/local is to handle non-base/non-packages stuff so you don't trash your system. Under OpenBSD, /usr/local is for packages, everything else should be under /opt or…

You took that python shell too literally. Ipython included a shell profile (you can still use it yourself) and https://xon.sh/ works just fine.
Post reply on HN