Unix Commands I Abuse Every Day • Hacker News Board

Re: Unix Commands I Abuse Every Day

#51

Post by ori_b » Thu, Sep 06, 2012, 3:00 AM UTC

veyron wrote: post #27

ori_b wrote: post #19

Earlier quoted context omitted.

Does it really matter that you're starting an extra process?

it doesnt matter for a file of size 1kb. For a file of size 10Gb, every process matters. For the downvoters: please time how long it takes to do something like `cat $file | awk '{print $1}' ` and `awk <$file '{print $1}'`

I was curious, so, here goes; 'foo' was a file of ~1G containing lines made up of of 999 'x's and one '\n'.

    $ ls -lh foo
    -rw-r--r-- 1 ori ori 954M Sep  5 22:57 foo

    $ time cat foo | awk '{print $1}' > /dev/null

    real	0m1.631s
    user	0m1.452s
    sys 	0m0.540s

    $ time awk  /dev/null 

    real	0m1.541s
    user	0m1.376s
    sys 	0m0.160s

This was run from a warm cache, so that the overhead of the extra IO from a pipe would dominate.

Re: Unix Commands I Abuse Every Day

#52

Post by ggchappell » Thu, Sep 06, 2012, 3:11 AM UTC

It should be noted that grep-dot only prints filenames if you give it more than one file.

Also, it skips blank lines. But of course that might be in the feature-not-a-bug category; and if you really want to see every line, there's always grep-quote-quote:

  grep '' *.txt

In any case, fun article. :-)

Re: Unix Commands I Abuse Every Day

#53

Post by BrandonM » Thu, Sep 06, 2012, 3:23 AM UTC

andrewvc wrote: post #2

My #1 abuse is xargs -n1. A lot of people like writing bash for loops, I will try and avoid that as much as possible, xargs -n1 is the bash equivalent of a call to 'map' in a functional language. For instance, let's say you want to create thumbnails of a bunch of jpegs: find images -name "*.jpg" | xargs -n1 -IF echo F F | sed -e 's/.jpg$/_thumb.jpg/' | xargs -n2 echo convert -geometry 200x Additionally, it's fully pa…

Why do that when a loop is clearer, safer, and shorter?

  find images -name '*.jpg' | while read jpg; do
      convert -geometry 200x "$jpg" "$(echo "$jpg" | sed 's/.jpg$/_thumb.jpg/')";
  done

This version works even when there are spaces in a filename, whereas yours will break.

Re: Unix Commands I Abuse Every Day

#54

Post by BrandonM » Thu, Sep 06, 2012, 3:29 AM UTC

Dobbs wrote: post #17

sillysaurus wrote: post #9

Earlier quoted context omitted.

Would anyone mind doing me a favor by explaining xargs in more detail? I've tried learning it a couple times but I always seem to forget the primary situations in which it's useful. Thank you in advance!

Xargs takes a newline separated list and maps the list to a command. find ./ -name '*.log' | xargs rm Finds all log files and map them to 'rm' commands. e.g. if it finds system.log and rails.log it will run the command `rm system.log rails.log`. xargs will automatically do things like break up very long lists into multiple command calls so that it doesn't exceed the maximum number of arguments a command can have. Oth…

Note that xargs actually takes a whitespace-delimited list. This often leads to problems when a filename has a space in it. To fix it, you should either use:

  find ... -print0 | xargs -0 ...

or:

  ... | xargs -d'\n' ...

Re: Unix Commands I Abuse Every Day

#55

Post by tlack » Thu, Sep 06, 2012, 3:29 AM UTC

3amOpsGuy wrote: post #26

I like this but i can't decide if it's technically abuse or not. The paste command will happily parse - (meaning read from stdin) multiple times, so for transposing a list into a table: file.txt: line1 line2 line3 line4 line5 line6 $ paste - - - Combine with the column command for pretty printing. I seem to find a use for this pretty frequently. I like the simplicity of this one but it's not very useful day to day: $…

Back in the crusty old days, FreeBSD used to take forever to install over the network, but would start an "emergency holographic shell" on pty4. The 'echo *' trick and various other shell built-ins were very useful for exploring the system before /bin and /usr/bin are populated.

Re: Unix Commands I Abuse Every Day

#56

Post by barrkel » Thu, Sep 06, 2012, 3:44 AM UTC

veyron wrote: post #27

ori_b wrote: post #19

Earlier quoted context omitted.

Does it really matter that you're starting an extra process?

it doesnt matter for a file of size 1kb. For a file of size 10Gb, every process matters. For the downvoters: please time how long it takes to do something like `cat $file | awk '{print $1}' ` and `awk <$file '{print $1}'`

Not exactly convincing:

    ~/desktop$ du -h c.dat
    11G     c.dat
    ~/desktop$ time cat c.dat | awk '{ print $1 }' > /dev/null
    
    real    0m53.997s
    user    0m52.930s
    sys     0m7.986s
    ~/desktop$ time  /dev/null
    
    real    0m53.898s
    user    0m51.074s
    sys     0m2.807s

cat CPU usage didn't exceed 1.6% at any time. The biggest cost is in redundant copying, so the more actual work you're doing on the data, the less and less it matters.

Re: Unix Commands I Abuse Every Day

#57

Post by sordina » Thu, Sep 06, 2012, 3:54 AM UTC

I've come to use the following command so often that I've written a script for it in my ~/bin - 'narrow':

    #!/bin/bash
    xargs -n 1 grep -l "$@"

This takes a list of files on stdin, then greps for the argument in all the files and spits out the matching files.

The perk is that it can be chained:

    find *.txt | narrow dog | narrow cat | narrow rabbit

This will find all the files that contain dog, cat, and rabbit.

Re: Unix Commands I Abuse Every Day

#58

Post by dustyleary » Thu, Sep 06, 2012, 4:43 AM UTC

ori_b wrote: post #51

veyron wrote: post #27

Earlier quoted context omitted.

it doesnt matter for a file of size 1kb. For a file of size 10Gb, every process matters. For the downvoters: please time how long it takes to do something like `cat $file | awk '{print $1}' ` and `awk <$file '{print $1}'`

I was curious, so, here goes; 'foo' was a file of ~1G containing lines made up of of 999 'x's and one '\n'. $ ls -lh foo -rw-r--r-- 1 ori ori 954M Sep 5 22:57 foo $ time cat foo | awk '{print $1}' > /dev/null real 0m1.631s user 0m1.452s sys 0m0.540s $ time awk /dev/null real 0m1.541s user 0m1.376s sys 0m0.160s This was run from a warm cache, so that the overhead of the extra IO from a pipe would dominate.

Both invocations take similiar amounts of "real" time because the task is IO-bound and it takes roughly 1.5s on your machine to read the file.

But if you add up the "user" and "sys" time in the cat example, you see that it took 1.992s of actual cpu-time... Which is actually about a 30% increase in cpu-time spent.

The perf decrease wasn't visible because you have multiple cores parallelizing the extra cpu-time, but it was there.

Re: Unix Commands I Abuse Every Day

#59

Post by philsnow » Thu, Sep 06, 2012, 5:22 AM UTC

dmayle wrote: post #25

jlgreco wrote: post #7

Simpler than using grep to highlight output is using ack with the --passthru option. (If you have ack anyway)

I use a custom function: function hl() { local R=$1; shift && egrep --color=always "|$R" $@; }

egregious use of color (uses 256-color term support):

function hl() { local R=''; while [ $# -gt 0 ]; do R="$R|$1"; shift; done; env GREP_COLORS="mt=38;5;$((RANDOM%256))" egrep --color=always $R; }

then do e.g.

whatever pipeline | hl foo bar baz | hl quux | hl '^.* frob.*$' | less -R

results in foo/bar/baz highlighted in one color, quux in another, and whole lines containing frob in another. hopefully the colors aren't indistinguishable from each other or from the terminal background :\

I use a somewhat similar setup in emacs, where a key binding adds the word under point to a syntax highlighting table, but the color is computed as the first 6 characters of the md5sum of the word.

Re: Unix Commands I Abuse Every Day

#60

Post by _ZeD_ » Thu, Sep 06, 2012, 5:35 AM UTC

BrandonM wrote: post #53

andrewvc wrote: post #2

My #1 abuse is xargs -n1. A lot of people like writing bash for loops, I will try and avoid that as much as possible, xargs -n1 is the bash equivalent of a call to 'map' in a functional language. For instance, let's say you want to create thumbnails of a bunch of jpegs: find images -name "*.jpg" | xargs -n1 -IF echo F F | sed -e 's/.jpg$/_thumb.jpg/' | xargs -n2 echo convert -geometry 200x Additionally, it's fully pa…

Why do that when a loop is clearer, safer, and shorter? find images -name '*.jpg' | while read jpg; do convert -geometry 200x "$jpg" "$(echo "$jpg" | sed 's/.jpg$/_thumb.jpg/')"; done This version works even when there are spaces in a filename, whereas yours will break.

false

    $ ls -Ql
    totale 4
    -rw-r--r-- 1 zed users 33 set  6 07:30 "    spaces    "
    $
    $ find -name '*spaces*' | while read text; do
        cat "$text";
    done
    cat: ./    spaces: File o directory non esistente
    $
    $ find -name '*spaces*' -print0 | xargs -0 cat
    while read is broken with spaces
    $