For the Love of Pipes • Hacker News Board

Re: For the Love of Pipes

#211

Post by nojvek » Tue, Jan 22, 2019, 6:27 PM UTC

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all.

Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects.

It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects and you can get the field you want by piping to `Get-Item` or sum the file sizes, or filter only directories. It’s very expressive once you’re manipulating streams of objects with properties.

Re: For the Love of Pipes

#212

Post by beachy » Tue, Jan 22, 2019, 6:27 PM UTC

masklinn wrote: post #196

Leace wrote: post #102

Earlier quoted context omitted.

It seems "white-space: pre-wrap" on code block would solve most of the problem. There is also additional "max-width" on the pre that I think is not needed.

That would break actual code snippet. What would solve most of the problems is HN actually implementing markdown instead of the current half-assed crap.

I would hate to see the day HN allowed any way to bold sections of text.

It's way more restful purveying a page of uniformly restrained text.

Re: For the Love of Pipes

#213

Post by jancsika » Tue, Jan 22, 2019, 6:27 PM UTC

jcelerier wrote: post #176

If you want to see what the endgame of this is when taking the reasoning to the maximum, look at visual dataflow languages such as Max/MSP, PureData, Reaktor, LabVIEW... Like always, simple stuff will be simple ( http://write.flossmanuals.net/pure-data/wireless-connections... ) and complicated stuff will be complicated ( https://ni.i.lithium.com/t5/image/serverpage/image-id/96294i... ). No silver bullet guys, sorry.…

> Like always, simple stuff will be simple (http://write.flossmanuals.net/pure-data/wireless-connections...)

That is not actually simple because the data is flowing across two completely different message passing paradigms. Many users of Max/MSP and Pd don't understand the rules for such dataflow, even though it is deterministic and laid out in the manual IIRC.

The "silver bullet" in Max/MSP would be to only use the DSP message passing paradigm. There, all objects are guaranteed to receive their input before they compute their output.

However, that would make a special case out of GUI building/looping/branching. For a visual language designed to accommodate non-programmers, the ease of handling larger amounts of complexity with impunity would not be worth the cost of a learning curve that excludes 99% of the userbase.

Instead, Pd and Max/MSP has the objects with thin line connections. They are essentially little Rube Goldberg machines that end up being about as readable. But they can be used to do branching/looping/recursion/GUI building. So users typically end up writing as little DSP as they can get away with then uses thin line spaghetti to fill in the rest. That turns out to be much cheaper than paying a professional programmer to re-implement their prototype at scale.

But that's a design decision in the language, not some natural law that visual programming languages are doomed to generate spaghetti.

Edit: clarification

Re: For the Love of Pipes

#214

Post by JdeBP » Tue, Jan 22, 2019, 6:29 PM UTC

elchief wrote: post #204

I wish there was an operator to allow a branch then join. Like send output to a lookup, do the lookup, join results Can do with named pipes, but an operator would be nice

GNU awk co-processes are another way to do this.

Re: For the Love of Pipes

#215

Post by EE84M3i » Tue, Jan 22, 2019, 6:32 PM UTC

crazygringo wrote: post #198

Pipes are awesome and infuriating. Sometimes they work great -- being able to dump from MySQL into gzip sending across the wire via ssh into gunzip and into my local MySQL without ever touching a file feels nothing short of magic... although the command/incantation to do so took quite a while to finally get right. But far too often they inexplicably fail. For example, I had an issue last year where piping curl to bun…

Perhaps your example was contrived, but why would you pipe into gzip instead of using transparent ssh compression?

Re: For the Love of Pipes

#216

Post by kemitche » Tue, Jan 22, 2019, 6:34 PM UTC

GuB-42 wrote: post #161

arendtio wrote: post #123

Earlier quoted context omitted.

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

IMHO, shell scripts are a minefield and if you want something readable and portable, this is also the wrong technology. They are convenient though. They are like the Excel macros of the UNIX world. Now back to the topic of "cat", which is a great example of why shell scripts are minefields. Replace "foo.txt" with a user supplied variable, let's call it "$F". It becomes cat $F | blah_blah... I mean cat "$F" | blah_bla…

> Now, if F='-n', second trap

You're not wrong, but I think it's worth pointing out that's a trap that comes up any time you exec another program, whether it's from shell or python. I can't reasonably expect `subprocess.run(["cat", somevar])` to work if `somevar = "-n"`.

(Now, obviously, I'm not going to "cat" from python, but I might "kubectl" or something else that requires care around the arguments)

Re: For the Love of Pipes

#217

Post by __MatrixMan__ » Tue, Jan 22, 2019, 6:34 PM UTC

jshen wrote: post #163

__MatrixMan__ wrote: post #149

Earlier quoted context omitted.

I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work. Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a…

I wonder if something like HTTP’s content negotiation is a good model for this.

That sounds reasonable, I'll look into it--thanks.

I was imagining an algorithm where each pipeline-aware utility can derive port numbers to use to talk/listen to its neighbors. I may be able to use http content negotiation wholesale in that context.

Re: For the Love of Pipes

#218

Post by Insanity » Tue, Jan 22, 2019, 6:42 PM UTC

alias_neo wrote: post #59

Insanity wrote: post #5

Earlier quoted context omitted.

I disagree - the pipe command is trivial at least to us whom have used linux (or unix) based systems for a long time. But to people who follow her and are rather new to this way of doing things, this could be a good read. Either way, it is nice to be reminded of the unix philosophy about chaining tools together for junior and senior alike. Edit: typo fixes

Fair enough. I'm a little surprised at the response. I didn't expect many people here aren't already intimately familiar with pipes, and for those that aren't *nix users, I'm not sure the content is of interest. I guess being honest around here is the wrong thing.

For what it's worth, I had no issue with your opinion and also didn't downvote either of your comments.

And I also didn't upvote the article, because it is indeed quite basic. Though apperantly enough people did find it interesting enough to make it to the frontpage ^^

Re: For the Love of Pipes

#219

Post by dahfizz » Tue, Jan 22, 2019, 6:46 PM UTC

beachy wrote: post #212

masklinn wrote: post #196

Earlier quoted context omitted.

That would break actual code snippet. What would solve most of the problems is HN actually implementing markdown instead of the current half-assed crap.

I would hate to see the day HN allowed any way to bold sections of text. It's way more restful purveying a page of uniformly restrained text.

How is that meaningfully different to italics in that regard?

Re: For the Love of Pipes

#220

Post by Leace » Tue, Jan 22, 2019, 6:47 PM UTC

quietbritishjim wrote: post #202

masklinn wrote: post #196

Earlier quoted context omitted.

That would break actual code snippet. What would solve most of the problems is HN actually implementing markdown instead of the current half-assed crap.

That would break any existing comments that happened to be using markdown syntax as punctuation. Although I suppose you could have a flag day for the changeover and format differently based on comment creation time. But I think the very limited formatting is just fine anyway. For the above comment as an example, I agree the code formatting looks awful, especially on mobile. But the version with >'s is ok, and I don't…

Conversations.im uses an interesting trick for rendering Markdown [0] - it leaves the syntax as is, so in the worst case you've got text with weird bold/italics, but the characters are 1:1 identical to what was sent.

[0]: Actually not Markdown but a subset but it's not important.