This reminds me of page 213 (175 in the book's numbering) of the Unix Haters Handbook, found at http://web.mit.edu/~simsong/www/ugh.pdf
Useful Unix commands for data science
61–70 of 108 posts
Re: Useful Unix commands for data science
#62A short and nice read is Unix for Poets by Kenneth Ward Church:
http://www.stanford.edu/class/cs124/kwc-unix-for-poets.pdf
Re: Useful Unix commands for data science
#63One of my favorite little tools that makes all these others better is pv -- PipeViewer. Use it any place in a pipeline to see a progress meter on stderr. Very handy when grepping through a bunch of big log files looking for stuff. Here is a quick strawman example: pv /data/*.log.gz | zgrep -c 'hello world' 241MiB 0:00:15 [15.8MiB/s] [==> ] 2% ETA 0:12:12
Shameless plug: here is a similar tool I wrote that prints not a progress bar but the contents flowing through the pipe, to help in debugging:
Re: Useful Unix commands for data science
#64Actually useful data science tips for unix users. Make all your commands 3x faster: export LC_ALL=C Actually use the 32 CPUs you paid for: sort --parallel=32 ... xargs -P32 ...
Could you expand on why export LC_ALL=C would "make all your commands 3x faster"?
Actually, it was more like 2000X[1] -- and I believe that it still stands as Brendan Gregg's biggest performance win.
[1] http://dtrace.org/blogs/brendan/2011/12/08/2000x-performance...
Re: Useful Unix commands for data science
#65One of my favorite little tools that makes all these others better is pv -- PipeViewer. Use it any place in a pipeline to see a progress meter on stderr. Very handy when grepping through a bunch of big log files looking for stuff. Here is a quick strawman example: pv /data/*.log.gz | zgrep -c 'hello world' 241MiB 0:00:15 [15.8MiB/s] [==> ] 2% ETA 0:12:12
BSD has a progress meter utility. It's called progress(1).
progress -zf /data/*.log.gz grep -c 'hello world'
progress -f /data/*.log.gz zgrep -c 'hello world'
You can also adjust buffer size, set the length for the time estimate (otherwise we have to fstat the input), and display progress to stderr instead of stdout.
Re: Useful Unix commands for data science
#66Earlier quoted context omitted.
If I needed to do this type of thing on 10 TB of data, it would probably take me longer to get the data to them than it would to just run it on my own hardware. Apparently there's a need for it, though, or it wouldn't exist.
Disclaimer: I work at Joyent, on Manta. This entire HN thread is a perfect example of why we built Manta. Lots of engineers/scientists/sysadmins/... already know how to (elegantly) process data using Unix and augmenting with scripts. Manta isn't about always needing to work on a 10TB dataset (you can), but about it being always available, and stored ready to go. I know we can't live without it for running our own sys…
Mark, is there any info on how I can figure out my monthly billing cost easily? Do I just need to sum the /user/reports/summary data for an estimate?
Re: Useful Unix commands for data science
#67Actually useful data science tips for unix users. Make all your commands 3x faster: export LC_ALL=C Actually use the 32 CPUs you paid for: sort --parallel=32 ... xargs -P32 ...
> Actually use the 32 CPUs you paid for:
gnu parallel FTW!
Re: Useful Unix commands for data science
#68Earlier quoted context omitted.
>I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger I concur with this recommendation. "The AWK Programming Language", at little over 100 pages, is a classic of programming language instruction. The book jumps right into use cases, it does not waste one's time. This book should be required reading for anyone contemplating writing a handbook on any programming language; my CS bookshelf would…
Sadly it seems very expensive now, $95 on Amazon...
Fortunately, a google search for "the awk programming language pdf" returns a link to this: http://books.cat-v.org/computer-science/awk-programming-lang...
It's the first result for me.
Re: Useful Unix commands for data science
#69AWK is worth learning completely. It hits a real sweet spot in terms of minimizing the number of lines of code needed to write useful programs in the world of quasi-structured (not quite CSV but not completely free form) data. You can learn the whole language and become proficient in an afternoon. I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger, though it seems to be listed for a hilariou…
Here is Kernighan's personal help file on AWK:
http://www.cs.princeton.edu/courses/archive/spr08/cos333/awk...
It deals with things he forgets or needs to remind himself of.
If you're interested in his other personal tutorials, they are here:
http://www.cs.princeton.edu/courses/archive/spr08/cos333/tut...
Re: Useful Unix commands for data science
#70Earlier quoted context omitted.
Yeah I wrote my own sum utility in Python... the syntax is just sum 1 or sum 2 for the column, with a -d delimiter flag. In retrospect I guess it could have been a one line awk script. But yeah if you are doing this kind of data-processing, it makes sense to have a hg/git repo of aliases and tiny commands that you sync around from machine to machine. You shouldn't have to write the sum more than once. Another useful…
I've never aliased it, but yes I use your 'hist' a lot. Useful for things like "categorise log errors" etc. Does everyone else edit command history, stacking up 'grep -v xxxx' in the pipeline to remove noise? If I'm working on a new pipeline, my normal workflow is something like: head file # See some representative lines head file | grep goodstuff head file | grep good stuff | grep -v badstuff head file | grep ... |…
> awk '{print $3}'
is the same as
> cut -f3 -d' '
cut is amazing for what it does. and most people know only the subset of awk that effectively _is_ cut anyway :D.