Useful Unix commands for data science
gregreda.com
Useful Unix commands for data science
1–10 of 108 posts
Re: Useful Unix commands for data science
#2I was hoping to see an article about some neat new utilities specifically tailored for doing advanced data analysis.
Instead this is a set of basic examples of bog-standard tools that every newbie *nix user should be already familiar with: cat, awk, head, tail, wc, grep, sed, sort, uniq
Re: Useful Unix commands for data science
#3I like slicing and dicing with awk, grep and friends too.
One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility?
echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command
echo "1\n2\n3\n4\n" | fold 0 + # should print 10
echo "1\n2\n3\n4\n" | fold 1 \* # should print 24
Re: Useful Unix commands for data science
#4Please be very careful doing math with bash and awk...
cat data.csv | awk -F "|" '{ sum += $4 } END { printf "%.2f\n", sum }'
Re: Useful Unix commands for data science
#5If you're interested in this general topic check out this Wikibook from John Rauser. He's a data scientist at Pinterest.
http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_U...
Re: Useful Unix commands for data science
#6I like slicing and dicing with awk, grep and friends too. One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility? echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command I suppose more generally you could have a 'fold initial op' and: echo "1\n2\n3\n4\n" | fold 0 + # should print 10 echo "1\n2\n3\n4\n" | fold 1 \* # should print…
Here's one way to do this with the standard "dc" (RPN calculator) utility:
echo "1\n2\n3\n+\n+\np\n" | dc -
> dc
1
2
3
+
+
p
6
Not sure how to automate this to sum 1000 values without needing to explicitly insert 999 + signs, though. Haven't explored dc in depth myself yet. There's probably some way to do it with a macro or something, but it may not be pretty.
Re: Useful Unix commands for data science
#7If this interests you, you should check out Joyents new Manta service which lets you do this type of thing on your data via their infrastructure. It's really cool.
Re: Useful Unix commands for data science
#8If you need them, Windows also has most of those tools somehow replicated in Powershell. For instance, the initial example can be replicated with:
Get-Content .\data.csv | %{[int]$total+=$_.Split('|')[3]; } ; Write-Host "$total"
Re: Useful Unix commands for data science
#9AWK is worth learning completely. It hits a real sweet spot in terms of minimizing the number of lines of code needed to write useful programs in the world of quasi-structured (not quite CSV but not completely free form) data. You can learn the whole language and become proficient in an afternoon.
I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger, though it seems to be listed for a hilariously high price on Amazon at the moment. Maybe try to pick up a used copy.
Re: Useful Unix commands for data science
#10I like slicing and dicing with awk, grep and friends too. One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility? echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command I suppose more generally you could have a 'fold initial op' and: echo "1\n2\n3\n4\n" | fold 0 + # should print 10 echo "1\n2\n3\n4\n" | fold 1 \* # should print…
You could do:
echo -e "1\n2\n3\n4" | paste -sd+ | bc
kind of cheating though! :-)