Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix tricks

mmb.pcb.ub.es

Post reply on HN
Print view View original

101–110 of 232 posts

Re: Unix tricks

#101

Post by maw »

etrain wrote: post #55

His last trick - compressed fie transfer without intermediate state: 'tar cz folder/ | ssh server "tar xz"' Can be pulled off with two flags to scp - and you get to see progress as a benefit! scp -Cr folder server:dest/

If security isn't a big consideration (read: you control both machines and the network), you go even faster with netcat.

On the receiving machine, in its destination directory:

    nc -l 6789 | tar xvf -
And on the sending machine, from its source directory:

    tar cvf - . | nc receiving-machine 6789
netcat varies a bit from distro to distro, so you may need to adjust these command lines a bit to get it to work.

Re: Unix tricks

#102

Post by gav »

niggler wrote: post #31
JoeAcchino wrote: post #21

Earlier quoted context omitted.

3) if all the lines of the 10+GB file are actually unique, wouldn't awk keep the whole file in RAM? For files larger than my RAM could this leave my system unresponsive because it's thrashing on swap?

The sort | uniq method literally needs to sort the file and pipe it to uniq, a far more memory-intensive operation than the single-pass awk check. You can write your own hash function in AWK if you think you may overstep memory, but of course you risk hash collisions. It's a tradeoff. I tried it on a 1U server with 24GB ram a few years ago and found that the sort was thrashing at the 10GB file size while AWK handled…

Or you can use `sort -u` and not have to pipe to `uniq`.

Re: Unix tricks

#103

Post by McUsr »

This was a nice very timely page sitting with my private repo in mercurial, and the other one at github…

I discovered help some months ago, and that was a great boon really. Like help "test" so I didn't have to go the rather large bash man page.

Here is a little shellscript for displaying a man page on Mac Os X (gman). (If you then click on on of the links on the man-page, it may pop up in your default browser).

  #!/bin/bash
  if [ $# -lt 1 ] ; then
  	echo "gman takes a man page, if found and formats it into html."
  	echo "Usage: gman [manfile]"
  	exit 2
  fi
  a=`man -aw $* |head -1`
  if test  x$a = x ; then
  	echo "Can't find $1"
  	exit 1
  fi
  # Figures out if it is a normal man page or something else (gz).
  b=`man -aw $* |head -1 |grep "gz"`
  echo $b
  if test  x$b = x ; then
  	groff -man $a -Thtml >|/tmp/tmp.html
  else
  	gzcat $b |groff -man -Thtml >|/tmp/tmp.html
  fi
  qlmanage -p /tmp/tmp.html >/dev/null 2&>1

Re: Unix tricks

#104

Post by Ensorceled »

AaronBBrown wrote: post #56
Ensorceled wrote: post #53

Weird, I've been using vi/elvis/vim/MacVim as my primary editor since 1984 and I hate vi key bindings for shell; I always use the emacs bindings.

I'm the opposite. Every time I log into a server that doesn't have my bashrc, I immediately have to `set -o vi` or else I'm useless.

That's why I love *nix ... options!

Re: Unix tricks

#105

Post by prakashk »

martinced wrote: post #88
globalpanic wrote: post #23

find . -name "file-wildcard" -exec "string" {} ";" -print is something I use a lot - plus xargs sometimes

I use variations on "find" so often that I've created several little commands "fij" (find in any java source), "fit" (find in any text / org file), "fix" (find in any XML file) etc. which I use all the time

You might want try `ack` then. (http://betterthangrep.com/).

On Ubuntu/Debian systems, it is packaged as `ack-grep`.

Re: Unix tricks

#106

Post by jerf »

ominous_prime wrote: post #93
jerf wrote: post #79

Earlier quoted context omitted.

In particular, scp is mindblowingly slow on lots of small files. I independently rediscovered the tar-pipe trick while sitting there watching scp laboriously copy thousands of 100-bytes so slowly I could count the files as they went by. That should not be possible, even at modem speeds. Fine for moving one file, OK for directories of very large files, not suitable for general usage where you might encounter a signifi…

Absolutely. Connection latency hits you the hardest, since each file is sent serially, and requires 2 (or 3 with -p) round trips in the protocol, and this is on top of an ssh tunnel with it's own overhead. I can't remember what my tests showed, but I have this inkling feeling that tar over ssh was far faster than rsync for an initial load, since there's no round trips required, but you lose some of possible rsync ben…

If my first tar attempt fails for some reason, but it made a lot of progress, I switch to rsync. Best of both worlds. This hasn't come up often enough for me to script it.

Re: Unix tricks

#107

Post by oftenwrong »

gcv wrote: post #65

Learn to use your shell's globbing features instead of overusing find. In zsh, you can do 'print -l /*.(c|cc|h|hh)' for example (I'm sure bash has an equivalent).

For me, the glob gets a lot of use.

Re: Unix tricks

#108

Post by nbouscal »

nicholassmith wrote: post #51
crusso wrote: post #20

The special bash command I'm most often asked about by shoulder surfers is !$. It substitutes the last argument in the previous command into the current one. For example: $ ls /some/long/path/somewhere/looking/around/ $ cd !$ cd /some/long/path/somewhere/looking/around/

I like that one and `!!` for the inevitable point I've forgotten to put sudo in before a command.

I find myself typing `sudo !!` on a daily basis. You'd think I'd eventually learn to remember to type sudo the first time, but nope.

Re: Unix tricks

#109

Post by ce4 »

inyourtenement wrote: post #97
stevvooe wrote: post #68

Earlier quoted context omitted.

You need a -z to activate compression. But, nonetheless, please always use rsync when copying host to host.

rsync is slow if the data is not already on the destination. tar over ssh is fast, and tar over socketpipe is even faster but not encrypted. I'm not aware of any attributes that tar doesn't preserve.

There's cryptcat if you need encryption. "bar" is also a nice little program if you like to see an ETA (c.f. http://clpbar.sourceforge.net/ )
Post reply on HN