Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking shell pipelines and the Unix “tools” philosophy

blog.plover.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#21

Post by tasogare »

tuldia wrote: post #18
tasogare wrote: post #15

Earlier quoted context omitted.

That’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution).

> That’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution). You are basically describing modern programming. Script Language (or scripting) is a programming language . And a…

You are overly pedantic on a detail point that doesn’t matter: yes shell scripting is technically a programming language, but my point is that it is a terrible one worth ditching for any non trivial task. Perl was created precisely more than 3 decades ago to address this problem. Nowadays there are alternatives such as Python, Powershell or even scripting wrapper for compiled languages (such as C#) that allow to do the same job very well, with less surprising behavior and that can be refactored later more easily.

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#22

Post by tingletech »

tuldia wrote: post #14
juped wrote: post #10

Earlier quoted context omitted.

Most people who use Unix directly build up some stuff in ~/bin (often a misnomer because it's shell scripts and not binaries, although mine is less of a misnomer than most because so much is in C rather than shell). The trick is to build them out of the standard portable components that exist everywhere. (This means, among other things, no #!/bin/bash.)

sed 's| no | not only |'

/bin/bash won't usually ship with a BSDish OS because of the license, so it is not generally portable to use bash-isms. (HPUX, IRIX, SunOS, Solaris, etc. I don't reckon would have had bash either)

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#23

Post by justinsaccount »

crystaldev wrote: post #11
justinsaccount wrote: post #3

'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems. However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient. I threw 144,000,000 'hello' and 'world' into a file: justin@box:~$ ls -…

All of your examples work in memory.

Not exactly. sort (at least GNU sort) will end up doing external merge sort on temporary files if you give it more data than you have memory. Which, if you give it 100GB of 5 different strings, ends up being a huge waste.

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#24

Post by vaingloriole »

yingw787 wrote: post #19
tuldia wrote: post #4

Thanks for this! Another nice thing about /usr/bin/time is the --verbose flag which gives: Command being timed: "ls" User time (seconds): 0.00 System time (seconds): 0.00 Percent of CPU this job got: 0% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size…

Wow this looks amazing! I didn’t know time could track all of those!

man getrusage

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#25

Post by tingletech »

tasogare wrote: post #15
amp108 wrote: post #7

Earlier quoted context omitted.

That's the whole point of shell scripting, to take a series of minimal programs and tie them together into something that does a more complex task. There's no reason to distrust a shell script simply because it is a script any more than there is to trust a binary simply because it's a binary.

That’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution).

I agree, and as TFA mentions, shell scripting is an "incoherent mess of composable paraphernalia" where you have to memorize a “weird mishmash of trivia” -- but if you have memorized it, then you can do some neat stuff without a so-called "real" language installed (that you have to upgrade and have to worry about if you have a modern version installed)

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#26

Post by tuldia »

justinsaccount wrote: post #9
tuldia wrote: post #6

Earlier quoted context omitted.

This is because in the first example you are invoking two programs. The first one sort the content of the file, the second count how many lines are equal. While in the awk example it is creating a hash table with all words and incrementing by the key and then printing. There is no sorting plus printing may be buffered.

Thanks for explaining my own comment to me.

Not explaining, trying to tell that you are comparing apples to oranges and making a conclusion based on that.

Also, you don't need to spawn a subshell nor feed sort via stdin in the first example :)

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#27

Post by vaingloriole »

tasogare wrote: post #21
tuldia wrote: post #18

Earlier quoted context omitted.

> That’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution). You are basically describing modern programming. Script Language (or scripting) is a programming language . And a…

You are overly pedantic on a detail point that doesn’t matter: yes shell scripting is technically a programming language, but my point is that it is a terrible one worth ditching for any non trivial task. Perl was created precisely more than 3 decades ago to address this problem. Nowadays there are alternatives such as Python, Powershell or even scripting wrapper for compiled languages (such as C#) that allow to do t…

I disagree with you wholeheartedly and without condition. Not only are you discounting how much time it takes to learn how to program effectively in a real 'glue' language you are high handed in ignoring the ubiquity, working archive and efficacy of a shell script. Not to mischaracterize but I find this type of attitude most frequently in 'lead' individuals with less than 10 years experience: typically 20's and 30's in age. I'm curious if this is your case?

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#28

Post by tuldia »

justinsaccount wrote: post #23
crystaldev wrote: post #11

Earlier quoted context omitted.

All of your examples work in memory.

Not exactly. sort (at least GNU sort) will end up doing external merge sort on temporary files if you give it more data than you have memory. Which, if you give it 100GB of 5 different strings, ends up being a huge waste.

Not only GNU sort, but also postgresql, mysql and many more...

Please, "huge waste"? How do you sort something that does not fit in memory?

Re: Benchmarking shell pipelines and the Unix “tools” philosophy

#29

Post by vaingloriole »

wahern wrote: post #16
skywhopper wrote: post #5

"What if Unix had less compositionality but I could use it with less memorized trivia? Would that be an improvement? I don't know." The answer is "no" here, because the alternative doesn't exist. Could it be created? Maybe in theory, but I suspect that the amount of stuff that you'd need to memorize (or learn to look up) to use it effectively would be about the same for any system that allowed a similar variety of wo…

There would be less trivia to memorize if the command behaviors and options were more consistent. You may not be able to achieve that at the edges, where new commands and options are added, but you can always go back and clean things up. For example, the cut(1) command is intended to do precisely what his f script does. But it's inconvenient because unlike many other commands it (1) doesn't obey $IFS and (2) the -d d…

Authority is the problem...not standardization and portability. Everyone is willing and able to tell you the best way to do your work if you use their tools. Straitjacketing implementation in the name of order is a surefire way to dissuade people from using your tools.
Post reply on HN