Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Unix Pipelines

prithu.xyz

Post reply on HN
Print view View original

291–300 of 388 posts

Re: The Beauty of Unix Pipelines

#291

Post by rcxdude »

scbrg wrote: post #276
jiggawatts wrote: post #141

Earlier quoted context omitted.

> Pipes are wonderful! It's wonderful only if compared to worse things, pretending that PowerShell is not a thing, and that Python doesn't exist. UNIX pipes are a stringly-typed legacy that we've inherited from the 1970s. The technical constraints of its past have been internalised by its proponents, and lauded as benefits . To put it most succinctly, the "byte stream" nature of UNIX pipes means that any command that…

The reality is that physically, it actually works like this: (process1,serialize) | (parse,process2,serialize) | ... But as soon as you involve the network or persistent storage, you need to do all that anyway . And one of the beauties is that the tools are agnostic to where the data comes from, or goes.

If you have network or persistent storage, those parse and serialise steps are better served by a consistent, typed, and common serialize and parse step instead of a different ad-hoc step for each part of the chain (especially if there are then parts of the chain which are just munging the output of one serialisation step so it can work with the parse stage of another).

Re: The Beauty of Unix Pipelines

#292

Post by junke »

enriquto wrote: post #277
junke wrote: post #274

Earlier quoted context omitted.

- This fails on badly formed PATH, or malicious PATH - When PATH contains entries that no longer exist / are not mounted, substitution with "sed" gives a wildcard that is not expanded, and eventually makes "file" report an error, which is not filtered out - If PATH contains entries that have spaces, the expansion is incorrect

More realistically, it also fails if you have directories on your path that are symlinks to other directories in the path. In that case their programs are doubly-counted. Anyways, if your PATH is malicious then you have worse problems than this silly script :)

I agree, but "more realistically"? those cases were pretty realistic I think. If you run the script on a box where you have users who can do whatever they want, at least the script should fail on bad inputs.

    PATH=rm -rf :/bin

Re: The Beauty of Unix Pipelines

#293

Post by boomlinde »

ssivark wrote: post #75
gorgoiler wrote: post #21

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in…

Serialized bytestreams do compose better than graphical applications. But that is setting a very low bar. For example, allowing passing around dicts/maps/json (and possibly other data structures) would already be a massive improvement. You know what might be even better — passing around objects you could interact with by passing messages (gasp! cue, Alan Kay). While posix and streams are nice (if you squint, files lo…

> For example, allowing passing around dicts/maps/json (and possibly other data structures) would already be a massive improvement.

Well, it is allowed. You can of course serialize some data into a structured format and pass it over a byte stream in POSIX to this end, and I think this is appropriate in some cases in terms of user convenience. Then you can use tools like xpath or jq to select subfields or mangle.

> (if you squint, files look like a poor man’s version of objects imposed on top of bytestreams)

If all you have is a hammer...

Re: The Beauty of Unix Pipelines

#295

Post by account42 »

tsimionescu wrote: post #255
coliveira wrote: post #77

Earlier quoted context omitted.

The shell works with spaces, you just need to be careful to quote every file name. The advantage of using file names without spaces is that you can avoid the quotes.

The shell (bash, anyway) has a weird mix of support and lack of support for spaces/lists/etc. For example, there is no way to store the `a b "c d"` in a regular variable in a way where you can then call something similar to `ls $VAR` and get the equivalent of `ls a b "c d"`. You can either get the behavior of `ls a b c d` or `ls "a b c d"`, but if you need `ls a b "c d"` you must go for an array variable with new syn…

Bash has arrays:

    var=( a b "c d" )
    ls "${var[@]}"
POSIX shell can get you what you want with eval:

    var='a b "c d"'
    eval "ls $var"
There is also one array variable in POSIX shells, the argument list:

    set -- a b "c d"
    ls "$@"

Re: The Beauty of Unix Pipelines

#296

Post by jrumbut »

laumars wrote: post #245
jrumbut wrote: post #84

Earlier quoted context omitted.

Also dd, which may be the disk destroyer but is also a great tool for binary file miracles. See one here: https://unix.stackexchange.com/questions/6852/best-way-to-re...

dd doesn’t example byte stream pipelines though ;) But yes, that’s the typical file / block management tool. Just remember to set a block size otherwise you’d suffer from worse performance than using cat

You're right! I was planning to tell a story about some bulk file changes using dd, but then I saw it was a giant "you had to be there" story so I posted with half a context.

Re: The Beauty of Unix Pipelines

#297

Post by jrumbut »

dredmorbius wrote: post #131
jrumbut wrote: post #84

Earlier quoted context omitted.

Also dd, which may be the disk destroyer but is also a great tool for binary file miracles. See one here: https://unix.stackexchange.com/questions/6852/best-way-to-re...

"data definition", from MVS JCL. https://news.ycombinator.com/user?id=dredmorbius

Yeah disk destroyer or data deleter or whatever is the old joke, not it's actual name. For those unaware, dd has often been used for stuff like overwriting a file with random noise as a homespun secure delete function, and it can do a lot of damage with small typos in the command.

Could have used a \s or something to clarify.

Re: The Beauty of Unix Pipelines

#298

Post by HeadsUpHigh »

enriquto wrote: post #258
jcranmer wrote: post #69

Earlier quoted context omitted.

There's a certain irony in responding to criticism of that which you're extolling by saying not to use it. And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.

> because spaces are a reasonable thing to put in a filename unless you're on a Unix system. Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.

I'm not going back to CamelCase or underscores for my normal day to day file naming. The problem with spaces only exists inside the IT world and it's something they should find a way around.

Re: The Beauty of Unix Pipelines

#299

Post by pjmlp »

pwdisswordfish2 wrote: post #280
pjmlp wrote: post #232

Earlier quoted context omitted.

Some people also insist to keep debuging with printf(), whereas the rest of the world is happily using InteliJ and friends.

I just use gdb and ald. I heard IntelliJ is a memory hog and requires a powerful computer.

InteliJ isn't the only option, hence why I mentioned "and friend".

Re: The Beauty of Unix Pipelines

#300

Post by nojito »

enriquto wrote: post #206
nojito wrote: post #108

Earlier quoted context omitted.

And yet they are absolutely useless if columns get swapped silently.

Everything breaks down if columns are swapped silently (or database records are renamed silently). What do you mean?

Not with Powershell.
Post reply on HN