Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix Commands I Wish I’d Discovered Years Earlier

spin.atomicobject.com

Post reply on HN
Print view View original

201–210 of 259 posts

Re: Unix Commands I Wish I’d Discovered Years Earlier

#201

Post by hobs »

WestCoastJustin wrote: post #113
Goladus wrote: post #109

Earlier quoted context omitted.

Great links, though I admit when I saw the OP I was expecting content like that and was pleasantly surprised that they really were "Unix commands I wish I'd discovered years earlier", and not the standard array of unix tips+tricks that you can get by without for awhile but as you become more advanced become second-nature. "Man ascii." The number of times that I wound up generating my own ascii table from whatever lan…

re: ascii tables You reminded me of something. There is actually a really cool and simple site called asciiflow [1], which I use all the time to draw diagrams for explaining things in email, etc. It's pretty cool, and was even submitted several times to HN [2, 3]. [1] http://www.asciiflow.com/#Draw [2] https://news.ycombinator.com/item?id=2847177 [3] https://news.ycombinator.com/item?id=3598177

Wow. This is really cool! Not only can you make the ascii drawings, but there is an integration with a service called ditaa (unaffiliated) that then turns your ascii drawings into actual images. I am keeping these. http://ditaa.sourceforge.net/

Re: Unix Commands I Wish I’d Discovered Years Earlier

#202

Post by whitlock »

csense wrote: post #171

My favorite is the parallel jobs feature of xargs. For example, say you want to run a script you wrote called process-video.sh to do some processing on all the video files in a directory (extracting audio to MP3, converting format, etc.). You want to use all 8 of your cores. You could write a Makefile and run it with -j9, or you can do this: find . -name "*.flv" | xargs -n 1 -P 9 ./process-video.sh This immediately f…

If you like xargs, but want more flexibility, I'd highly suggest GNU parallel. Such flexibility includes running jobs on multiple computers, running intensive command using all available CPU's (like xargs -P), and creating unique scripts to handle multiple parameters.

http://www.gnu.org/software/parallel/man.html

Re: Unix Commands I Wish I’d Discovered Years Earlier

#203

Post by ilikepi »

Typhon wrote: post #198
ilikepi wrote: post #140

Earlier quoted context omitted.

FWIW, the double-star recursive globbing was added in Bash 4.0. My reading of the manual[1] suggests it is disabled by default, and can be enabled by setting the 'globstar' shell option. I believe it's enabled by default in zsh. [1]: https://www.gnu.org/software/bash/manual/bash.html#The-Shopt...

I just tried it on bash 4.2.25, and it worked, even though I just learned about it. Maybe not every OS enables the same default options for Bash (I'm using Lubuntu 12.10 at the moment)

Yep, that is a fair point. I should have qualified my statement about bash's defaults, since many OSs include their own custom start-up files that may modify the defaults.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#204

Post by jfb »

mturmon wrote: post #167
arnsholt wrote: post #128

Earlier quoted context omitted.

It means that whatever you put into the ssh process's STDIN gets sent to the STDIN of the remote process. For example you can do this: tar cj $bigdir | ssh host 'tar xj' That creates a tar.bz2 archive on your machine, sends it over ssh to the remote host, where it's unpacked. Ad hoc compressed and encrypted file transfers made easy.

That's a good illustration, but rsync, if available, would generally be preferred for that use case: rsync -az $bigdir host:$bigdir

Sure, of course. But I've used that tar|ssh combination so much that it was the first example of using the rsh-like stdout/in pairing that came to mind. As much as I bitch about Unix and POSIX and horrific shell brain damage, I still reach for crazy shell pipelines when I have data manipulation tasks up to some pretty large scale n.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#206

Post by jfb »

webreac wrote: post #153
dmckeon wrote: post #97

Earlier quoted context omitted.

This is a bit of a pet peeve of mine, but cd/tar should likely be joined with && - in other words, in what directory do you want tar to (not) run if cd fails? So consider: % (cd /foo && tar cpf - .) | ssh bar '(cd /baz && tar xpf -)' or explore something like: % rsync -aSHx --update --delete -e ssh /foo/ user@bar:baz/.

GNU tar has evolved and can use simpler syntax: % tar cp -C /foo . | ssh bar tar xp -C /baz

It would be an interesting project to look at people's shell usage and try to determine when they started using Unix. I have loads of Irixisms that have only recently been crowded out of my head by Darwin BSDism.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#207

Post by jfb »

sopooneo wrote: post #74
jfb wrote: post #58

Earlier quoted context omitted.

The single most useful thing about ssh to me is that it connects stdout/in across the channel (this is behavior inherited from rsh). This allows for e.g.: % (cd /foo; tar cpf - .) | ssh bar \(cd /baz\; tar xpf -\) Or: % cat ~/.ssh/id_rsa.pub | ssh bar tee -a .ssh/authorized_keys

Forgive my ignorance, but I don't know what what it means to connect "across the channel", but it sounds important. Do you know a link that explains the concept? I've failed to Google it.

Are you familiar with stdin/stdout/stderr on Unix and how they interact? What ssh does that I find so powerful is ensure that what you put into stdout on one side of a pipe flows out of stdin on the process that ssh starts on the remote host.

Of course, ssh also gives you the output of the remote process, so you can put an ssh command in the middle of a pipeline. My day to day life no longer involves interacting with lots of remote Unix systems, but when it did, it was really, really useful.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#208

Post by sikhnerd »

oftenwrong wrote: post #68
WestCoastJustin wrote: post #47

Earlier quoted context omitted.

One of my favorite things about unix is that you can chain commands together with pipe (|). Say for example that you have a example data file that looks like this: # Deflection Col-Force Beam-Force 0.000 0 0 0.001 104 51 0.002 202 101 0.003 298 148 0.0031 104 149 0.004 289 201 0.0041 291 209 0.005 104 250 0.010 311 260 0.020 104 240 You can chain commands together to quickly get ball park figures. I do this all the t…

Yes, there are some ways to simplify. You can get rid of the "UUOC". Instead of cat data.txt | grep -v '#' you can do: grep -v '#' data.txt or you can simplify further by moving the grepping into awk, so that the full command becomes awk '!/#/{print $2}' data.txt | sort -n | uniq -c sort -n is good to use for sorting numbers. You could probably pull the sort | uniq -c into awk somehow, but I have already demonstrated…

You can pre-sort as greyfade mentions, or do it within awk itself:

  awk '!/#/ {a[$2]++}END{for (i in a) print a[i],i | "sort -nk1"}' data.txt
 1 0
 1 202
 1 289
 1 291
 1 298
 1 311
 4 104

Re: Unix Commands I Wish I’d Discovered Years Earlier

#210

Post by willthames »

0x45696e6172 wrote: post #162
jfb wrote: post #58

Earlier quoted context omitted.

The single most useful thing about ssh to me is that it connects stdout/in across the channel (this is behavior inherited from rsh). This allows for e.g.: % (cd /foo; tar cpf - .) | ssh bar \(cd /baz\; tar xpf -\) Or: % cat ~/.ssh/id_rsa.pub | ssh bar tee -a .ssh/authorized_keys

ssh-copy-id is made for exactly this: ssh-copy-id - install your public key in a remote machine’s authorized_keys

the tip does work on OS X though, which doesn't have ssh-copy-id (by default, at least)
Post reply on HN