Benchmarking shell pipelines and the Unix “tools” philosophy

Post by chriswitts » Wed, Jan 08, 2020, 6:15 AM UTC

juped wrote: post #10

Neil44 wrote: post #2

I got excited when I saw the 'f' and 'count' commands, but they're just scripts he has on his system. Like doing grep 'plover' blah.log | cut -d ' ' -f 11 | sort | uniq -c | sort -n. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system, but they are nice.

Most people who use Unix directly build up some stuff in ~/bin (often a misnomer because it's shell scripts and not binaries, although mine is less of a misnomer than most because so much is in C rather than shell). The trick is to build them out of the standard portable components that exist everywhere. (This means, among other things, no #!/bin/bash.)

Post by inimino » Wed, Jan 08, 2020, 7:05 AM UTC

boyter wrote: post #47

tuldia wrote: post #4

Thanks for this! Another nice thing about /usr/bin/time is the --verbose flag which gives: Command being timed: "ls" User time (seconds): 0.00 System time (seconds): 0.00 Percent of CPU this job got: 0% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size…

Can anyone comment why you can only use the verbose flag if you use the full path of time? time -v ls does not work but /usr/bin/time -v ls does? I don't have enough knowledge of either linux applications or bash to know whats happening to cause this.

Post by tuldia » Wed, Jan 08, 2020, 7:30 AM UTC

cperciva wrote: post #44

tuldia wrote: post #26

Earlier quoted context omitted.

Not explaining, trying to tell that you are comparing apples to oranges and making a conclusion based on that. Also, you don't need to spawn a subshell nor feed sort via stdin in the first example :)

He's comparing apples to oranges and reaching the conclusion that... yes, apples and oranges are different things. He's quite aware of this, and even points out the tradeoff -- `sort | uniq -c` still works if your dataset doesn't fit into RAM.

Post by tuldia » Wed, Jan 08, 2020, 7:57 AM UTC

boyter wrote: post #47

tuldia wrote: post #4

Thanks for this! Another nice thing about /usr/bin/time is the --verbose flag which gives: Command being timed: "ls" User time (seconds): 0.00 System time (seconds): 0.00 Percent of CPU this job got: 0% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size…

Can anyone comment why you can only use the verbose flag if you use the full path of time? time -v ls does not work but /usr/bin/time -v ls does? I don't have enough knowledge of either linux applications or bash to know whats happening to cause this.

  \time -v ls

Post by dredmorbius » Wed, Jan 08, 2020, 8:48 AM UTC

crystaldev wrote: post #11

justinsaccount wrote: post #3

'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems. However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient. I threw 144,000,000 'hello' and 'world' into a file: justin@box:~$ ls -…

All of your examples work in memory.

    #!/usr/bin/gawk -f
    { x[NR] = $1 }
    END {
        rc = asort(x)
        j=0
        for(i in x) {
            if( x[i] "" == x[i-1] "" ) freq[j]++
     else {
         j++
         elem[j] = x[i]
         freq[j] = 1
     }
        }
        for(j in elem) {
            printf( "%6i %s\n", freq[j], elem[j])
        }
    }

    #!/usr/bin/gawk -f
    { x[$1]++ }
    END { for(i in x ) printf( "%6i %s\n", x[i], i ) }

    for i in {1..2000}; do echo $((RANDOM%10)); done  > data