Starts off with unnecessary use of cat, e.g., cat file | awk 'cmds'.
One can simply do awk 'cmds' file.
Useful Unix commands for data science
11–20 of 108 posts
Re: Useful Unix commands for data science
#12I like slicing and dicing with awk, grep and friends too. One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility? echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command I suppose more generally you could have a 'fold initial op' and: echo "1\n2\n3\n4\n" | fold 0 + # should print 10 echo "1\n2\n3\n4\n" | fold 1 \* # should print…
Just for grins:
$ alias sum="xargs | tr ' ' '+' | bc"
$ echo -e "1\n2\n3\n" | sum
6Re: Useful Unix commands for data science
#13I was hoping to see an article about some neat new utilities specifically tailored for doing advanced data analysis. Instead this is a set of basic examples of bog-standard tools that every newbie *nix user should be already familiar with: cat, awk, head, tail, wc, grep, sed, sort, uniq
>"tools that every newbie nix user should be already familiar with"
The key word is should... you might be surprised how many "not newbie" nix users are not aware of those commands or how using them in this fashion. Specially awk.
Re: Useful Unix commands for data science
#14A commenter on the article pointed out the "Useless use of cat".
What most users probably don't realize is that the redirection can be anywhere on the line, not just at the beginning. Putting an input redirection at the beginning of the command can make the data flow clearer: from the input file, through the command, to stdout:
Re: Useful Unix commands for data science
#15Actually useful data science tips for unix users.
Make all your commands 3x faster:
export LC_ALL=C
Actually use the 32 CPUs you paid for:
sort --parallel=32 ...
xargs -P32 ...Re: Useful Unix commands for data science
#16I'm surprised there was no mention of cut -d. It's good for simple stuff where you don't need all of awk.
Re: Useful Unix commands for data science
#17AWK is worth learning completely. It hits a real sweet spot in terms of minimizing the number of lines of code needed to write useful programs in the world of quasi-structured (not quite CSV but not completely free form) data. You can learn the whole language and become proficient in an afternoon. I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger, though it seems to be listed for a hilariou…
I recall there was a pointer to an old great AWK tutorial some time ago - smth along the lines 'how to approach awk language....' - anyone kept the link?
Re: Useful Unix commands for data science
#18The "Text Processing" category of this list of unix utilities is also helpful: http://en.wikipedia.org/wiki/List_of_Unix_programs
BashReduce is a pretty cool application of many of these utilities.
Re: Useful Unix commands for data science
#19In some cases, (command line) Perl will actually process piped text faster than awk or even sed. I'm not sure about arithmetic though.
Re: Useful Unix commands for data science
#20Please be very careful doing math with bash and awk... cat data.csv | awk -F "|" '{ sum += $4 } END { printf "%.2f\n", sum }' From that command, it's unclear whether the sum will be accurate, it depends on the inputs and on the precision of awk. See (D.3 Floating-Point Number Caveats): http://www.delorie.com/gnu/docs/gawk/gawk_260.html
Please be very careful doing math with bash and awk...
I don't see how that's more true for awk than it is for any other programming language. Awk uses double precision floating point for all numeric values, which isn't a horrible choice for a catch-all numeric type.