Unix Commands I Wish I’d Discovered Years Earlier • Hacker News Board

Re: Unix Commands I Wish I’d Discovered Years Earlier

#61

Post by WestCoastJustin » Tue, Sep 10, 2013, 4:53 PM UTC

fsckin wrote: post #28

One of the more useful bits of ssh is not mentioned: remotely running commands. Example: ssh username@host "echo $HOSTNAME && sudo somecommand && cat somecommand.log" There's probably a better way to do this, but in a pinch I can fix a problem on dozens of machines just by altering the host string.

Yeah, before I was heavily into automation (i.e. puppet) this was my go to command. Use a for loop and ssh to run things on a group of systems.

  for host in h-abc h-def h-ghi h-jkl h-mno h-pqr h-stu h-vwx h-yza; 
    do 
    ssh -l root $host "hostname; echo "1.1.1.1 server" >> /etc/hosts"; 
  done

This would iterate over hosts (h-abc .. h-yza) and run the commands "hostname; echo "1.1.1.1 server" >> /etc/hosts". Or maybe use a for loop and scp to push the files out and then run md5 to verify they are correct. It was a hack but useful for a small set of machines.

ps. if you are interested in learning about puppet, I have put together a screencast about it @ http://sysadmincasts.com/episodes/8-learning-puppet-with-vag...

Re: Unix Commands I Wish I’d Discovered Years Earlier

#62

Post by res0nat0r » Tue, Sep 10, 2013, 4:55 PM UTC

ck2 wrote: post #6

I want a command that counts files in a tree like du does sizes but without having to pipe find through wc which is crazy for hundred thousand files. Can't they just directly access inodes for high speed counting?

There really isn't any way around this. The first pass over your data is going to be slow while the inode data is examined, but it will be cached for the next pass. If you do this before that data gets removed from the cache it will be much faster.

  imac:~
  $ time find . | wc -l
  419191

  real	0m37.865s
  user	0m0.385s
  sys	0m2.611s
  imac:~
  $ time find . | wc -l
  419195

  real	0m10.503s
  user	0m0.330s
  sys	0m1.430s

Re: Unix Commands I Wish I’d Discovered Years Earlier

#63

Post by D9u » Tue, Sep 10, 2013, 4:56 PM UTC

laumars wrote: post #35

D9u wrote: post #33

Earlier quoted context omitted.

I have xxd on my FreeBSD setup. uname -rms FreeBSD 9.1-RELEASE-p19 amd64 Although the xxd man page doesn't have the BSD label, and is from 1996, so I'm not sure if xxd only recently found its way into FBSD. This manual page documents xxd version 1.7 AUTHOR (c) 1990-1997 by Juergen Weigert

It maybe in ports, but it's not part of the base install: $ xxd xxd: Command not found. $ uname -rms FreeBSD 9.1-RELEASE amd64 edit: fixed typo where I put xdd instead of xxd

I didn't add xxd on my local FBSD (installed last night) setup, and my remote FBSD shell has xxd installed by default...

    $ uname -rms
    FreeBSD 9.1-RELEASE-p7 amd64

    $ xxd -v
    xxd V1.10 27oct98 by Juergen Weigert

Perhaps different builds have different things included?

Re: Unix Commands I Wish I’d Discovered Years Earlier

#64

Post by thejosh » Tue, Sep 10, 2013, 4:58 PM UTC

One of my favourite traceroute replacements, that is installed by default on Ubuntu: mtr !

Re: Unix Commands I Wish I’d Discovered Years Earlier

#65

Post by laumars » Tue, Sep 10, 2013, 4:59 PM UTC

D9u wrote: post #63

laumars wrote: post #35

Earlier quoted context omitted.

It maybe in ports, but it's not part of the base install: $ xxd xxd: Command not found. $ uname -rms FreeBSD 9.1-RELEASE amd64 edit: fixed typo where I put xdd instead of xxd

I didn't add xxd on my local FBSD (installed last night) setup, and my remote FBSD shell has xxd installed by default... $ uname -rms FreeBSD 9.1-RELEASE-p7 amd64 $ xxd -v xxd V1.10 27oct98 by Juergen Weigert Perhaps different builds have different things included?

No, it installs as part of vim: https://news.ycombinator.com/item?id=6361210

Re: Unix Commands I Wish I’d Discovered Years Earlier

#66

Post by D9u » Tue, Sep 10, 2013, 5:02 PM UTC

laumars wrote: post #50

pessimizer wrote: post #46

Earlier quoted context omitted.

s/xdd/xxd/g

Sorry, that was a typo when posting on here. I did originally run it as xxd: root@primus:/usr/ports # xxd xxd: Command not found. edit: It doesn't show up in my ports either: root@primus:/usr/ports # make search name=xxd Port: textproc/xxdiff Moved: Date: 2013-07-26 Reason: Has expired: Depends on Qt 3.x Port: textproc/xxdiff-scripts Moved: Date: 2013-07-26 Reason: Has expired: Depends on Qt 3.x Obviously xxdiff is a…

    $ whereis xxd
    xxd: /usr/local/bin/xxd /usr/local/man/man1/xxd.1.gz

I don't see xxd anywhere in my ports tree.

Haha... What a trip. This is the sort of thing that I'll end up wasting half a day trying to get an answer to.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#67

Post by philsnow » Tue, Sep 10, 2013, 5:04 PM UTC

Keyframe wrote: post #55

hannibal5 wrote: post #23

Unix and Common Lisp are similar in many ways: 1. Both are operating systems 2. Both are conceptually simple. 3. There is always nice command that does what you want but you somehow forget it. I can't believe that Common Lisp has just 900+ symbols but I routinely forget some of them when programming.

My favorite part is when you implement something you could've had running in a minute or so, but instead developed it for hours or days and it's still not as good as the existing solution. I often get that with C and libraries. There has to be a better way to search through existing stuff which is good and proven, not just for C of course.

Haskell has hoogle [0], that lets you search by type signature. If you already know roughly what kind of function you want, it's a lot easier to sift through a tiny handful of (or even a single) functions that match a given type signature.

ISTR Forth had something similar, or maybe I'm just thinking that in Forth, words are almost uniquely defined by their stack annotations.

[0] http://www.haskell.org/hoogle/

Re: Unix Commands I Wish I’d Discovered Years Earlier

#68

Post by oftenwrong » Tue, Sep 10, 2013, 5:05 PM UTC

WestCoastJustin wrote: post #47

Kurtz79 wrote: post #39

I know it's not exactly an unknown command, but I didn't know about "sort" until last week. It's freaking fast and convenient, sorts hadoop reduce results like a champ.

One of my favorite things about unix is that you can chain commands together with pipe (|). Say for example that you have a example data file that looks like this: # Deflection Col-Force Beam-Force 0.000 0 0 0.001 104 51 0.002 202 101 0.003 298 148 0.0031 104 149 0.004 289 201 0.0041 291 209 0.005 104 250 0.010 311 260 0.020 104 240 You can chain commands together to quickly get ball park figures. I do this all the t…

Yes, there are some ways to simplify. You can get rid of the "UUOC". Instead of

     cat data.txt | grep -v '#'

you can do:

     grep -v '#' data.txt

or you can simplify further by moving the grepping into awk, so that the full command becomes

      awk '!/#/{print $2}' data.txt | sort -n | uniq -c

sort -n is good to use for sorting numbers. You could probably pull the sort | uniq -c into awk somehow, but I have already demonstrated the full extent of my awk knowledge. One of these days I will get around to actually learning it.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#69

Post by zipperhead » Tue, Sep 10, 2013, 5:05 PM UTC

shawkinaw wrote: post #54

codegeek wrote: post #49

I love "tail -f" to see realtime updates on a file that I am writing to!!

If you don't want to clog up your terminal, try 'less' and then hit 'F' to follow.

Better yet use 'less +F'. It's as easy to remember as tail -f.

The advantage of using less here is that you can easily interrupt the tailing (ctrl-c). For example, to search. Then just hit 'F' again to restart tailing. But now your search pattern will be highlighted as the data streams by.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#70

Post by philsnow » Tue, Sep 10, 2013, 5:06 PM UTC

WestCoastJustin wrote: post #61

fsckin wrote: post #28

One of the more useful bits of ssh is not mentioned: remotely running commands. Example: ssh username@host "echo $HOSTNAME && sudo somecommand && cat somecommand.log" There's probably a better way to do this, but in a pinch I can fix a problem on dozens of machines just by altering the host string.

Yeah, before I was heavily into automation (i.e. puppet) this was my go to command. Use a for loop and ssh to run things on a group of systems. for host in h-abc h-def h-ghi h-jkl h-mno h-pqr h-stu h-vwx h-yza; do ssh -l root $host "hostname; echo "1.1.1.1 server" >> /etc/hosts"; done This would iterate over hosts (h-abc .. h-yza) and run the commands "hostname; echo "1.1.1.1 server" >> /etc/hosts". Or maybe use a fo…

make a small change:

  for host in ... ; do ssh -l root $host "..." & done; wait

and it's very parallel. All that key exchange takes a while, so it's not ridiculously fast, but it's pretty good. If you've got ControlMaster set up already to be per-host, this might very well be very fast the second time.