Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking shell pipelines and the Unix “tools” philosophy

blog.plover.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#61

Post by justinsaccount »

dredmorbius wrote: post #59
crystaldev wrote: post #11

Earlier quoted context omitted.

All of your examples work in memory.

Even working in memory, there are different efficiencies for different methods. Awk includes an asort() function which can sort an array, such that it would be possible to create a similar process entirely within awk to the sort | uniq -c pipeline: #!/usr/bin/gawk -f { x[NR] = $1 } END { rc = asort(x) j=0 for(i in x) { if( x[i] "" == x[i-1] "" ) freq[j]++ else { j++ elem[j] = x[i] freq[j] = 1 } } for(j in elem) { pri…

With super small dataset sizes like that it'll fit in the L2 cache and can behave differently. I did test this though, and for 20,000 items generated with your loop:

sort | uniq -c takes .017s (fastest out of a few runs)

the awk command I used above takes .013s

A trivial implementation I have in go takes .08s

Additionally, using this 'protos' file which is 1,000,000 lines of tcp,udp,icmp:

  $ time (sort protos|uniq -c)
     5915 icmp
   332003 tcp
   662082 udp

  real 0m0.232s
  user 0m0.739s
  sys 0m0.100s

  $ # fixed to count 'lines' and not the first column, which makes it faster.
  $ time awk '{lines[$0]++} END {for (l in lines) printf("%s %d\n", l, lines[l])}'
so yes, I do test my assumptions.

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#62

Post by nibbula »

jessant wrote: post #8
skywhopper wrote: post #5

"What if Unix had less compositionality but I could use it with less memorized trivia? Would that be an improvement? I don't know." The answer is "no" here, because the alternative doesn't exist. Could it be created? Maybe in theory, but I suspect that the amount of stuff that you'd need to memorize (or learn to look up) to use it effectively would be about the same for any system that allowed a similar variety of wo…

That is what the author says in the next paragraph. > I don't know. I rather suspect that there's no way to actually reach that hypothetical universe.

I'm already living in that universe, but with even more composability.

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#63

Post by dredmorbius »

justinsaccount wrote: post #61
dredmorbius wrote: post #59

Earlier quoted context omitted.

Even working in memory, there are different efficiencies for different methods. Awk includes an asort() function which can sort an array, such that it would be possible to create a similar process entirely within awk to the sort | uniq -c pipeline: #!/usr/bin/gawk -f { x[NR] = $1 } END { rc = asort(x) j=0 for(i in x) { if( x[i] "" == x[i-1] "" ) freq[j]++ else { j++ elem[j] = x[i] freq[j] = 1 } } for(j in elem) { pri…

With super small dataset sizes like that it'll fit in the L2 cache and can behave differently. I did test this though, and for 20,000 items generated with your loop: sort | uniq -c takes .017s (fastest out of a few runs) the awk command I used above takes .013s A trivial implementation I have in go takes .08s Additionally, using this 'protos' file which is 1,000,000 lines of tcp,udp,icmp: $ time (sort protos|uniq -c)…

I was suggesting @crystaldev may not have.

And, more significantly, and as you've confirmed, not all in-memory processing is equivalent. There are faster and slower all-in-memory algorithms and implementations

Hrm... Maybe a bogounique implementation might be appropriate here....

(By analogy to bogosort: https://en.wikipedia.org/wiki/Bogosort)

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#64

Post by Lerc »

chriswitts wrote: post #51
juped wrote: post #10

Earlier quoted context omitted.

Most people who use Unix directly build up some stuff in ~/bin (often a misnomer because it's shell scripts and not binaries, although mine is less of a misnomer than most because so much is in C rather than shell). The trick is to build them out of the standard portable components that exist everywhere. (This means, among other things, no #!/bin/bash.)

You shouldn't be using `#!/bin/bash` but rather `#!/usr/bin/env bash` instead.

I know this but usually don't do it. Been burnt by it too. Reprogramming your brain is hard.

[edit] Would there be any negative consequences to having an automated process go through and change it? Maybe the size difference might cause some issues with some things doing black magic with data in the file.

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#65

Post by m463 »

CGamesPlay wrote: post #46
m463 wrote: post #41

the ACTUAL benchmark should be: 1) start timer 2) start deciding which commands to pipeline together 3) run the commands 4) stop timer a lot of times the decision is the long pole. in this authors case it included: 5) try a couple more variants of steps 2 and 3 6) write a blog post :)

This is a good point about the second half of the article (compositonality), but the author started the article by saying this was a command he "sometimes runs", presumably indicating he has it saved somewhere.

My commands of this type are usually retrieved using control-r
Post reply on HN