I'm surprised there was no mention of cut -d. It's good for simple stuff where you don't need all of awk.
and paste - quick and simple for working with columns
Useful Unix commands for data science
21–30 of 108 posts
Re: Useful Unix commands for data science
#22Actually useful data science tips for unix users. Make all your commands 3x faster: export LC_ALL=C Actually use the 32 CPUs you paid for: sort --parallel=32 ... xargs -P32 ...
Could you expand on why
export LC_ALL=C
Re: Useful Unix commands for data science
#23I like slicing and dicing with awk, grep and friends too. One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility? echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command I suppose more generally you could have a 'fold initial op' and: echo "1\n2\n3\n4\n" | fold 0 + # should print 10 echo "1\n2\n3\n4\n" | fold 1 \* # should print…
If you consider the use of dc/bc as in the other solutions to be cheating, you can use unary-encoded integers...
alias sum='xargs -I{} sh -c "head -c {} Re: Useful Unix commands for data science
#24Actually useful data science tips for unix users. Make all your commands 3x faster: export LC_ALL=C Actually use the 32 CPUs you paid for: sort --parallel=32 ... xargs -P32 ...
Could you expand on why export LC_ALL=C would "make all your commands 3x faster"?
Gnu grep is or was very slow with the UTF-8 locale. Not sure about other commands, perhaps anything that processes text, awk and sed maybe?
Re: Useful Unix commands for data science
#25A commenter on the article pointed out the "Useless use of cat". What most users probably don't realize is that the redirection can be anywhere on the line, not just at the beginning. Putting an input redirection at the beginning of the command can make the data flow clearer: from the input file, through the command, to stdout: (This only works for simple commands; you can't do `< file if blah; then foo; else bar; fi…
"Useless use of cat" is one of those boring pedantic comments that makes me cringe. Who cares? It's usually much more straightforward to build a pipeline from left to right, particularly for people who are just learning this stuff.
Re: Useful Unix commands for data science
#26Actually useful data science tips for unix users. Make all your commands 3x faster: export LC_ALL=C Actually use the 32 CPUs you paid for: sort --parallel=32 ... xargs -P32 ...
Could you expand on why export LC_ALL=C would "make all your commands 3x faster"?
If your text manipulation programs are locale-aware, they may be interpreting the input as a multibyte encoding, and need to do a lot more work in preprocessing to get semantically correct operation. For example, a Unicode-aware grep may understand more forms of equivalence, similarly for sorting. See e.g. http://en.wikipedia.org/wiki/Unicode_equivalence
With the C locale, text is more or less treated as plain bytes.
Re: Useful Unix commands for data science
#27I like slicing and dicing with awk, grep and friends too. One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility? echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command I suppose more generally you could have a 'fold initial op' and: echo "1\n2\n3\n4\n" | fold 0 + # should print 10 echo "1\n2\n3\n4\n" | fold 1 \* # should print…
Yeah I wrote my own sum utility in Python... the syntax is just sum 1 or sum 2 for the column, with a -d delimiter flag. In retrospect I guess it could have been a one line awk script. But yeah if you are doing this kind of data-processing, it makes sense to have a hg/git repo of aliases and tiny commands that you sync around from machine to machine. You shouldn't have to write the sum more than once.
Another useful one is "hist" which is sort | uniq -c | sort -n -r.
Re: Useful Unix commands for data science
#28I like slicing and dicing with awk, grep and friends too. One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility? echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command I suppose more generally you could have a 'fold initial op' and: echo "1\n2\n3\n4\n" | fold 0 + # should print 10 echo "1\n2\n3\n4\n" | fold 1 \* # should print…
paste -sd+|bcRe: Useful Unix commands for data science
#29Starts off with unnecessary use of cat, e.g., cat file | awk 'cmds'. One can simply do awk 'cmds' file.
I know purists always complain about unnecessary cats, but I always find it useful to start with "head" or "tail" in the first position to figure out my pipeline, and then replace it with cat when it's all working.
And if the extra cat is actually making a measurable difference, maybe that's a good signal that it's time to rewrite it in C.
Re: Useful Unix commands for data science
#30Actually useful data science tips for unix users. Make all your commands 3x faster: export LC_ALL=C Actually use the 32 CPUs you paid for: sort --parallel=32 ... xargs -P32 ...
Also export MAKEOPTS=-j33