Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For the Love of Pipes

blog.jessfraz.com

Post reply on HN
Print view View original

281–290 of 323 posts

Re: For the Love of Pipes

#281

Post by rabidrat »

icedchai wrote: post #171
arendtio wrote: post #123

Earlier quoted context omitted.

In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…

If the order is your concern, you can also put the <read at the beginning of the line. <file grep x works the same as: cat file | grep x

I've been using unix for 25 years and I did not know that.

Re: For the Love of Pipes

#282

Post by flexer2 »

One of my most used tools with pipes on the Mac is `pbcopy`, which copies whatever you're piping to the clipboard.

Copying your public key somewhere? `cat ~/.ssh/id_rsa.pub | pbcopy`

Pretty-printing some JSON you copied from a log file? `pbpaste | jq . | pbcopy`

It's one of those little tools I use daily and don't think about much, but it's incredibly useful.

Re: For the Love of Pipes

#283

Post by saagarjha »

flexer2 wrote: post #282

One of my most used tools with pipes on the Mac is `pbcopy`, which copies whatever you're piping to the clipboard. Copying your public key somewhere? `cat ~/.ssh/id_rsa.pub | pbcopy` Pretty-printing some JSON you copied from a log file? `pbpaste | jq . | pbcopy` It's one of those little tools I use daily and don't think about much, but it's incredibly useful.

And of course, its cousin pbpaste is also quite useful: pbpaste | clang -x c -; ./a.out

Re: For the Love of Pipes

#284

Post by hnick »

Jyaif wrote: post #96
alkonaut wrote: post #82

I love the idea of simple things that can be connected in any way. I'm not so much a fan of "everything is a soup of bytes with unspecified encoding and unknown formatting". It's an abstraction that held up quite well, but its starting to show its age.

100% agree. Having to extract information with regular expressions is a waste of time. If the structure of the data was available, you would have type safety / auto-completion. You could even have GUIs to compose programs.

Having GUIs compose programs seems antithetical to the idea of shell scripts which are often thrown together quickly to get things done. Personally, I view shell scripting as a "good enough" and if you need more structure then you change your tools.

Re: For the Love of Pipes

#285

Post by [deleted] »

nojvek wrote: post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

[deleted]

Re: For the Love of Pipes

#286

Post by anthk »

noir_lord wrote: post #264
kradroy wrote: post #262

I built an entire prototype ML system/pipeline using shell scripts that glued together two python scripts that did some heavy lifting not easily reproduced. I got the whole thing working from training to prediction in about 3 weeks. What I love about Unix shell commands is that you simply can't abstract beyond the input/output paradigm. You aren't going to create classes, types classes, tests, etc. It's not possible…

My backup system at work is mostly bash scripts and some pipes. If you write them cleanly they don’t suck and crucially for me, bash today works basically the same way as it did 10 years ago and likely in 10 years, that static nature is a big damn win. I sometimes wish language vendors would just say ‘this language is complete, all future work will be bug fixes and libraries’ a static target for anything would be nic…

- My IRC notification system is a shell script with entr, notify-send and dunst.

- My mail setup uses NMH, everything is automated. I can mime-encode a directory and send the resulting mail in a breeze.

- GF's photos from IG are being backed up with a python script and crontab. Non IG ones are geotagged too with a script. I just fire up some cli GPS tools if we hike some mountain route, and gpscorrelate runs on the GPX file.

- Music is almost everything chiptunes, I felt interesent on any mainstream music since 2003-4. I mirror a site with wget and it's done. If they offered rsync...

- Hell, even my podcasts are being fetch via cron(8).

- My setup is CWM/cli based, except for mpv, emulators, links+ and vimb for JS needed sites. Noice is my fm, or the pure shell. find(1) and mpg123/xmp generate my music playlist. Street View is maybe the only service I use on vimb...

The more you automate, the less tasks you need to do. I am starting to avoid even taskwarrior/timew, because I am almost task free as I don't have to track a trivial https://github.com/pickfire/spt is everything I need.

Also, now I can't stand any classical desktop, I find bloat on everything.

Re: For the Love of Pipes

#287

Post by flukus »

wglb wrote: post #194
fooblat wrote: post #182

Earlier quoted context omitted.

Well find, cal, and ls don't take input at all. And vi (well vim) in fact does. If you invoke it with a - for the file name argument, it will read standard input into the buffer. I can't comment on emacs as I don't use it much.

Ah, correct on that point. What about standard output? That is, can vi be used in a pipe?

With some redirection magic it can be made to, but I find that magic is always too much stuffing around to make it useful. Inside scripts though I like to pipe to vim and have it output to a file:

  pipeline | vim - +"file $mytempfile"
Then if you quit vim without saving $mytempfile won't exist, if you save and quit it will and further processing can be done. I've got a view scripts like this to to things like viewing csv files after being piped through column, after saving they're converted back to csv.

Re: For the Love of Pipes

#288

Post by flukus »

nojvek wrote: post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

> Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects.

Unix pipes can pipe objects, binary data, text, json encoded data, etc.

The problem is it adds a lot of complication and simply doesn't offer much in practice, so text is still widely used and binary data in a few specific cases.

Re: For the Love of Pipes

#289

Post by anthk »

jgtrosh wrote: post #24
benj111 wrote: post #17

Earlier quoted context omitted.

Chmod +x /dev/null (havent tried above, not sure I recommend that you do)

Well you can't read either from /dev/null, and I don't think that's just a question of permissions. I'm pretty sure it's impossible to get /dev/null to behave like an executable.

http://peetm.com/blog/?p=55

Re: For the Love of Pipes

#290

Post by anthk »

jfhufl wrote: post #160
benj111 wrote: post #10

Why isn't the pipe a construct that has caught on in 'proper' languages?

Hmmm, well that's why I like Ruby and other languages with functional approaches. Method chaining and blocks are very similar to pipes to me. cat /etc/passwd | grep root | awk -F: '{print $3}' ruby -e 'puts File.read("/etc/passwd").lines.select { |line| line.match(/root/) }.first.split(":")[2]' A little more verbose, but the idea is the same. https://alvinalexander.com/scala/fp-book/how-functional-prog...

awk -F: '/root/ {print $3}' < /etc/passwd
Post reply on HN