Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For the Love of Pipes

blog.jessfraz.com

Post reply on HN
Print view View original

141–150 of 323 posts

Re: For the Love of Pipes

#141

Post by zorga »

Hello71 wrote: post #134
arendtio wrote: post #123

Earlier quoted context omitted.

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

This is a very silly way of writing it though. grep|sed can almost always be replaced with a simple awk: awk '/^x/ { sub("a", "b"); print $2; }' foo.txt. This way, the whole command fits on one line. If it doesn't, put your awk script in a separate file and simply call it with "awk -f myawkscript foo.txt".

And you've utterly missed the point. His version is better than your suggestion, it's cleaner and more readable and built from simpler parts. Banging grep and sed together is far more common than digging into awk.

Re: For the Love of Pipes

#142

Post by mweberxyz »

roelb wrote: post #128

Fully agree, pipes are awesome, only downside is the potential duplicate serialization/deserialization overhead. Streams in most decent languages closely adhere to this idea. I especially like how node does it, in my opinion one of the best things in node. Where you can simply create cli programs that have backpressure the same as you would work with binary/file streams, while also supporting object streams. process.…

Node streams are excellent, but unfortunately don't get as much fanfare as Promises/async+await. A number of times I have gotten asked "how come my node script runs out of memory" -- due to the dev using await and storing the entirety of what is essentially streaming data in memory in between processing steps.

Re: For the Love of Pipes

#144

Post by AdmiralAsshat »

arendtio wrote: post #123
yaakushi wrote: post #85

I'm probably nitpicking, but if you're using cat to pipe a single file into the sdtin of another program, you most likely don't need the cat in the first place, you can just redirect the file to the process' stdin. Unless, of course, you're actually concatenating multiple files or maybe a file and stdin together. Disclaimer: I do cat-piping myself quite a bit out of habit, so I'm not trying to look down at the author…

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

That's actually very readable. I'm now regretting that I hadn't seen this about 3 months ago--I recently left a project that had a large number of shell scripts I had written or maintained for my team. This probably would've made it much easier for the rest of the team to figure out what the command was doing.

Re: For the Love of Pipes

#145

Post by yesenadam »

Hello71 wrote: post #134
arendtio wrote: post #123

Earlier quoted context omitted.

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

This is a very silly way of writing it though. grep|sed can almost always be replaced with a simple awk: awk '/^x/ { sub("a", "b"); print $2; }' foo.txt. This way, the whole command fits on one line. If it doesn't, put your awk script in a separate file and simply call it with "awk -f myawkscript foo.txt".

>sub("a", "b");

That should be gsub, shouldn't it? (sub only replaces the first occurrence)

Re: For the Love of Pipes

#146

Post by adem666 »

Unix's philosophy of “do one thing well” and “expect the output of every program to become the input to another” is living with "microservices" in nowadays.

Re: For the Love of Pipes

#147

Post by mamcx »

benj111 wrote: post #10

Why isn't the pipe a construct that has caught on in 'proper' languages?

Others have show languages have some kind of pipe support, but not exactly as the shell.

The shell have the weird(?) behavior of ONE input and TWO outputs (stdout, stderr).

Also, can redirect in both directions. I think a language to be alike pipes, it need each function to be alike:

    fun open(...) -> Result(Ok,Err)
and have the option of not only chain the OK side but the ERR:

    open("file.txt") |> print !!> raise |> print
exist something like this???

Re: For the Love of Pipes

#149

Post by __MatrixMan__ »

jimktrains2 wrote: post #71
pytester wrote: post #62

Earlier quoted context omitted.

The core critique - that everything is stringly typed - still holds pretty well though. >The receiving and sending processes must use a stream of bytes. Any object more complex than a byte cannot be sent until the object is first transmuted into a string of bytes that the receiving end knows how to reassemble. This means that you can’t send an object and the code for the class definition necessary to implement the ob…

To be fair, the same critisim could be used for a socket? I think the issue is that some people want pipes to be something magical that connects their software, not a dumb connection between them.

I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work.

Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a proof of concept:

https://unix.stackexchange.com/a/495338/146169

Do you think this is a reasonable way to achieve the magic that some users want in their pipelines? Or are ancient Unix gods going to smite me for tampering with the functional consistency of tools by making their behavior different in different contexts?

Re: For the Love of Pipes

#150

Post by testvox »

jenscow wrote: post #116
Jyaif wrote: post #96

Earlier quoted context omitted.

100% agree. Having to extract information with regular expressions is a waste of time. If the structure of the data was available, you would have type safety / auto-completion. You could even have GUIs to compose programs.

I hear what you're saying. However, how can you ensure the output type of one program matches the input type of another?

Allow programs to specify the type of data they can consume and the type of the data they emit. This is how powershell does it (using the dotnet type system).
Post reply on HN