Benchmarking shell pipelines and the Unix “tools” philosophy
blog.plover.com
Benchmarking shell pipelines and the Unix “tools” philosophy
1–10 of 66 posts
Re: Benchmarking shell pipelines and the Unix “tools” philosophy
#2I got excited when I saw the 'f' and 'count' commands, but they're just scripts he has on his system. Like doing grep 'plover' blah.log | cut -d ' ' -f 11 | sort | uniq -c | sort -n. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system, but they are nice.
Re: Benchmarking shell pipelines and the Unix “tools” philosophy
#3'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems.
However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient.
I threw 144,000,000 'hello' and 'world' into a file:
justin@box:~$ ls -lh words
-rw-r--r-- 1 justin justin 824M Jan 7 15:21 words
justin@box:~$ wc -l words
144000000 words
justin@box:~$ time (sort justin@box:~$ time awk '{words[$1]++} END {for (w in words) printf("%s %d\n", w, words[w])}' Re: Benchmarking shell pipelines and the Unix “tools” philosophy
#4Thanks for this!
Another nice thing about /usr/bin/time is the --verbose flag which gives:
Command being timed: "ls"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 0%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 1912
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 112
Voluntary context switches: 1
Involuntary context switches: 1
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Re: Benchmarking shell pipelines and the Unix “tools” philosophy
#5"What if Unix had less compositionality but I could use it with less memorized trivia? Would that be an improvement? I don't know."
The answer is "no" here, because the alternative doesn't exist. Could it be created? Maybe in theory, but I suspect that the amount of stuff that you'd need to memorize (or learn to look up) to use it effectively would be about the same for any system that allowed a similar variety of work to be accomplished. If you are willing to trade off functionality for simplicity, then sure, it can be done. You can get it today by just not using all these tools at all, I suppose.
Re: Benchmarking shell pipelines and the Unix “tools” philosophy
#6'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems. However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient. I threw 144,000,000 'hello' and 'world' into a file: justin@box:~$ ls -…
This is because in the first example you are invoking two programs. The first one sort the content of the file, the second count how many lines are equal.
While in the awk example it is creating a hash table with all words and incrementing by the key and then printing.
There is no sorting plus printing may be buffered.
Re: Benchmarking shell pipelines and the Unix “tools” philosophy
#7I got excited when I saw the 'f' and 'count' commands, but they're just scripts he has on his system. Like doing grep 'plover' blah.log | cut -d ' ' -f 11 | sort | uniq -c | sort -n. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system, but they are nice.
That's the whole point of shell scripting, to take a series of minimal programs and tie them together into something that does a more complex task. There's no reason to distrust a shell script simply because it is a script any more than there is to trust a binary simply because it's a binary.
Re: Benchmarking shell pipelines and the Unix “tools” philosophy
#8"What if Unix had less compositionality but I could use it with less memorized trivia? Would that be an improvement? I don't know." The answer is "no" here, because the alternative doesn't exist. Could it be created? Maybe in theory, but I suspect that the amount of stuff that you'd need to memorize (or learn to look up) to use it effectively would be about the same for any system that allowed a similar variety of wo…
That is what the author says in the next paragraph.
> I don't know. I rather suspect that there's no way to actually reach that hypothetical universe.
Re: Benchmarking shell pipelines and the Unix “tools” philosophy
#9'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems. However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient. I threw 144,000,000 'hello' and 'world' into a file: justin@box:~$ ls -…
This is because in the first example you are invoking two programs. The first one sort the content of the file, the second count how many lines are equal. While in the awk example it is creating a hash table with all words and incrementing by the key and then printing. There is no sorting plus printing may be buffered.
Thanks for explaining my own comment to me.
Re: Benchmarking shell pipelines and the Unix “tools” philosophy
#10I got excited when I saw the 'f' and 'count' commands, but they're just scripts he has on his system. Like doing grep 'plover' blah.log | cut -d ' ' -f 11 | sort | uniq -c | sort -n. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system, but they are nice.
Most people who use Unix directly build up some stuff in ~/bin (often a misnomer because it's shell scripts and not binaries, although mine is less of a misnomer than most because so much is in C rather than shell). The trick is to build them out of the standard portable components that exist everywhere. (This means, among other things, no #!/bin/bash.)