Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for data science

gregreda.com

Post reply on HN
Print view View original

71–80 of 108 posts

Re: Useful Unix commands for data science

#71

Post by Osmium »

gnosis wrote: post #2

I was hoping to see an article about some neat new utilities specifically tailored for doing advanced data analysis. Instead this is a set of basic examples of bog-standard tools that every newbie *nix user should be already familiar with: cat, awk, head, tail, wc, grep, sed, sort, uniq

Don't forget there are only ever going to be more unix newbies in the world. It's not like they're a dying breed. There are more people than ever who have never been exposed to unix tools who might benefit from them (myself included several years ago).

Re: Useful Unix commands for data science

#72

Post by jnazario »

daemon13 wrote: post #17
minimax wrote: post #9

AWK is worth learning completely. It hits a real sweet spot in terms of minimizing the number of lines of code needed to write useful programs in the world of quasi-structured (not quite CSV but not completely free form) data. You can learn the whole language and become proficient in an afternoon. I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger, though it seems to be listed for a hilariou…

I recall there was a pointer to an old great AWK tutorial some time ago - smth along the lines 'how to approach awk language....' - anyone kept the link?

i wrote one a long time ago:

http://linuxgazette.net/67/nazario.html

i still use a buttload of awk for data science type uses.

Re: Useful Unix commands for data science

#73

Post by mcavage »

res0nat0r wrote: post #66
mcavage wrote: post #56

Earlier quoted context omitted.

Disclaimer: I work at Joyent, on Manta. This entire HN thread is a perfect example of why we built Manta. Lots of engineers/scientists/sysadmins/... already know how to (elegantly) process data using Unix and augmenting with scripts. Manta isn't about always needing to work on a 10TB dataset (you can), but about it being always available, and stored ready to go. I know we can't live without it for running our own sys…

Mark, is there any info on how I can figure out my monthly billing cost easily? Do I just need to sum the /user/reports/summary data for an estimate?

Yeah that's why we generate ~/reports for you every hour - that's what our billing runs off of. I know there's an internal "turn that into daily $ script" somebody wrote -- we'll get that put out as a sample job.

Re: Useful Unix commands for data science

#74

Post by toupeira »

ibotty wrote: post #70
jbert wrote: post #51

Earlier quoted context omitted.

I've never aliased it, but yes I use your 'hist' a lot. Useful for things like "categorise log errors" etc. Does everyone else edit command history, stacking up 'grep -v xxxx' in the pipeline to remove noise? If I'm working on a new pipeline, my normal workflow is something like: head file # See some representative lines head file | grep goodstuff head file | grep good stuff | grep -v badstuff head file | grep ... |…

> awk '{print $3}' is the same as > cut -f3 -d' ' cut is amazing for what it does. and most people know only the subset of awk that effectively _is_ cut anyway :D.

Not exactly, awk will consume all whitespace while cut will split on each individual space character, and not on newlines and tabs.

Re: Useful Unix commands for data science

#75

Post by hafabnew »

reyan wrote: post #62

A short and nice read is Unix for Poets by Kenneth Ward Church: http://www.stanford.edu/class/cs124/kwc-unix-for-poets.pdf

I actually was about to post this -- this guide is great. As an undergraduate, this was what was given to us to help demonstrate above-introductory command line tools/pipes.

Re: Useful Unix commands for data science

#76

Post by mjn »

mcavage wrote: post #56
jlgaddis wrote: post #40

Earlier quoted context omitted.

If I needed to do this type of thing on 10 TB of data, it would probably take me longer to get the data to them than it would to just run it on my own hardware. Apparently there's a need for it, though, or it wouldn't exist.

Disclaimer: I work at Joyent, on Manta. This entire HN thread is a perfect example of why we built Manta. Lots of engineers/scientists/sysadmins/... already know how to (elegantly) process data using Unix and augmenting with scripts. Manta isn't about always needing to work on a 10TB dataset (you can), but about it being always available, and stored ready to go. I know we can't live without it for running our own sys…

Wow, this looks great. My ideal cloud-computing platform is basically something like xargs -P or GNU parallel, but with the illusion that I'm running it on a machine with infinite CPU cores and RAM (charged for usage, of course). I was spoiled early on by having once had something almost like that, via a very nice university compute cluster, where your data was always available on all nodes (via NFS), and you just prefixed your usual Unix commands with a job-submit command, which did the magic of transparently running stuff wherever it wanted to run it. Apart from the slight indirection of using the job-submit tool, it almost succeeded in giving the illusion of ssh-ing into a single gazillion-core big-iron machine, which is more or less the user experience I want. But I haven't found a commercial offering where I can get an account on a big Unix cluster and just get billed for some function of my (disk space, CPU usage, RAM usage) x time.

Cloud services are amazing in a lot of ways, but so far I've found them much more heavyweight for the use-case of running ad-hoc jobs from the Unix command line. You don't really want to write Hadoop code for exploratory data analysis, and even managing a little fleet of bashreduce+EC2 instances that get spun up and down on demand is error-prone and tedious, turning me more into the cluster administrator rather than a user, which is what I'd rather be. Admittedly it's possible that could be abstracted out better in the case where you don't mind latency: I often don't mind if my jobs queue up for a few minutes, which would mean a tool could spin up EC2 instances behind the scenes and then tear them down without me noticing. But I haven't found anything that does that transparently yet, and Manta looks like a more direct implementation of the "illusion of running on an N-core machine for arbitrary N" idea that seems in the same cost ballpark. Definitely going to do some experimentation here, to see if 2010s technology will enable me to keep using a 1970s-era data-processing workflow.

Re: Useful Unix commands for data science

#77

Post by tzs »

jbert wrote: post #53
aidos wrote: post #47

Earlier quoted context omitted.

Sometimes I like to start with cat so I can easily swap for zcat when changing to gripped input.

Agree. Or actually I start with a 'head -100' so I don't handle too much data in my pipeline until it's ready.

I'm old fashioned, so use "sed 100q" instead of the newer "head -100". It saves a keystroke, too.

There are enough variations in ways to do things on Unix that I've sometimes wondered about how easy it would be to identify a user by seeing how they accomplish a common task.

For instance, I noticed at one place I worked that even though everyone used the same set of options when doing a "cpio -p", everyone had their own order they wrote them. Seeing one "cpio -p" command was sufficient to tell which of the half dozen of us had done the command.

I think I'm the only one where I work who uses "sed Nq" instead of "head -N", so that would fingerprint me.

Re: Useful Unix commands for data science

#78

Post by gnosis »

bcantrill wrote: post #64
gnosis wrote: post #22

Earlier quoted context omitted.

Could you expand on why export LC_ALL=C would "make all your commands 3x faster"?

Actually, it was more like 2000X[1] -- and I believe that it still stands as Brendan Gregg's biggest performance win. [1] http://dtrace.org/blogs/brendan/2011/12/08/2000x-performance...

According to the comments in that thread, this issue was fixed in GNU grep 2.7 (my system currently has grep 2.14 on it, so this must have been some time ago).

Re: Useful Unix commands for data science

#79

Post by robinson-wall »

gnosis wrote: post #2

I was hoping to see an article about some neat new utilities specifically tailored for doing advanced data analysis. Instead this is a set of basic examples of bog-standard tools that every newbie *nix user should be already familiar with: cat, awk, head, tail, wc, grep, sed, sort, uniq

Indeed. I'm going to start calling myself a data scientist now, instead of a sysadmin. See if I can't get a raise.

Re: Useful Unix commands for data science

#80

Post by wasd »

I've been using Ubuntu for about a year now and although I feel comfortable doing a lot of things with the CL, I'm not sure if I really know enough about *nix. I wish there was a was a website with the 20-30 most useful unix commands and very clear language as to what they do with examples. Although, I've used all the tools in this post, I still enjoyed the use of example.
Post reply on HN