Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for data science

gregreda.com

Post reply on HN
Print view View original

91–100 of 108 posts

Re: Useful Unix commands for data science

#91

Post by Sprint »

jbert wrote: post #51
chubot wrote: post #27

Earlier quoted context omitted.

Yeah I wrote my own sum utility in Python... the syntax is just sum 1 or sum 2 for the column, with a -d delimiter flag. In retrospect I guess it could have been a one line awk script. But yeah if you are doing this kind of data-processing, it makes sense to have a hg/git repo of aliases and tiny commands that you sync around from machine to machine. You shouldn't have to write the sum more than once. Another useful…

I've never aliased it, but yes I use your 'hist' a lot. Useful for things like "categorise log errors" etc. Does everyone else edit command history, stacking up 'grep -v xxxx' in the pipeline to remove noise? If I'm working on a new pipeline, my normal workflow is something like: head file # See some representative lines head file | grep goodstuff head file | grep good stuff | grep -v badstuff head file | grep ... |…

Bundling expressions into regular expressions can be handy, for example "grep -Ev '(thisbot|thatbot|bingbot|bongbot)'" instead many single grep pipes.

Re: Useful Unix commands for data science

#92

Post by koralatov »

sleepydog wrote: post #44
mds wrote: post #29

Earlier quoted context omitted.

I know purists always complain about unnecessary cats, but I always find it useful to start with "head" or "tail" in the first position to figure out my pipeline, and then replace it with cat when it's all working. And if the extra cat is actually making a measurable difference, maybe that's a good signal that it's time to rewrite it in C.

You can do with simple IO redirection. For example, the arbitrary pipeline $ cat data.txt | awk '{ print $2+$4,$0 }'|sort|sed '/^0/d' can be written as $

Some people prefer the first, longer-winded way because it's more explicit. To some --- myself included --- it makes more sense because it explicitly breaks each function into a seperate steps; I'm explicitly telling the system to print the contents of data.txt rather than implictly doing so. I'll happily type those five extra characters for that additional clarity.

Re: Useful Unix commands for data science

#93

Post by lotsofcows »

tzs wrote: post #77
jbert wrote: post #53

Earlier quoted context omitted.

Agree. Or actually I start with a 'head -100' so I don't handle too much data in my pipeline until it's ready.

I'm old fashioned, so use "sed 100q" instead of the newer "head -100". It saves a keystroke, too. There are enough variations in ways to do things on Unix that I've sometimes wondered about how easy it would be to identify a user by seeing how they accomplish a common task. For instance, I noticed at one place I worked that even though everyone used the same set of options when doing a "cpio -p", everyone had their o…

Stoll's "The Cuckoo's Egg" contains a bit of detective work based around the hacker's switch style.

Re: Useful Unix commands for data science

#94

Post by lotsofcows »

gwu78 wrote: post #88

"Imagine you have a 4.2GB CSV file." " All you need... is the sum of all values in one particular column." In that case, if speed was paramount, I'd use Kona or kdb. Unquestionably, k is the best tool for that particular job.

Really? In a recent test of a whole bunch of languages, scripting, compiled and JVM (but not Kona or kdb), our awk test was beaten only by C. awk was so far ahead, its run time beat other's compile + run time.

Re: Useful Unix commands for data science

#95

Post by dpatru »

kamaal wrote: post #84

>>Writing a script in python/ruby/perl/whatever would probably take a few minutes and then even more time for the script to actually complete. Thankfully you can also write a Perl one liner. Which most of the times is far powerful than awk.

sum the 0th field: perl -lane '$a += $F[0]; END{ print $a; }'

Re: Useful Unix commands for data science

#96

Post by jrn »

mcavage wrote: post #56
jlgaddis wrote: post #40

Earlier quoted context omitted.

If I needed to do this type of thing on 10 TB of data, it would probably take me longer to get the data to them than it would to just run it on my own hardware. Apparently there's a need for it, though, or it wouldn't exist.

Disclaimer: I work at Joyent, on Manta. This entire HN thread is a perfect example of why we built Manta. Lots of engineers/scientists/sysadmins/... already know how to (elegantly) process data using Unix and augmenting with scripts. Manta isn't about always needing to work on a 10TB dataset (you can), but about it being always available, and stored ready to go. I know we can't live without it for running our own sys…

I know manta, has default software packaged, but is it possible to install your own like ghci, or julia? Or is that something that needs to be brought in as an asset. This isn't necessarily a feature request, just trying to figure out how it works. https://apidocs.joyent.com/manta/compute-instance-software.h...

Re: Useful Unix commands for data science

#97

Post by dredmorbius »

zeidrich wrote: post #8

If you need them, Windows also has most of those tools somehow replicated in Powershell. For instance, the initial example can be replicated with: Get-Content .\data.csv | %{[int]$total+=$_.Split('|')[3]; } ; Write-Host "$total"

Or you can actually use the Linux commands by installing Cygwin. Pretty much my first conscious action when I wake up stranded on a desert Windows system.

Re: Useful Unix commands for data science

#98

Post by merlincorey »

mjn wrote: post #35
makmanalp wrote: post #32

Everyone forgets the brilliant and sometimes crazy BSD ones: - Column: Create columns / tables from input data - tr: substitute / delete chars - join: like a database join, but for text files - comm: like diff, but you can use it programmatically to choose if a line is in one file, or another, or both. - paste: put file lines side-by-side - rs: reshape arrays - jot: generate random or sequence data - expand: replace…

Looks like 6 of those 8 are in GNU coreutils as well (and therefore can be assumed present on just about any modern Unix). 'rs' and 'jot' are the two missing from most default Linux installs. On Debian you can install them via the packages 'rs' and 'athena-jot'.

'jot' is pretty sweet, especially for creating ranges for iteration and random numerical data for testing arguments and such.

Check out the man page for a few snippets: http://www.unix.com/man-page/FreeBSD/1/jot/

It is the older, more flexible uncle of gnu's 'seq' command: http://administratosphere.wordpress.com/2009/01/23/using-bsd...

Re: Useful Unix commands for data science

#99

Post by pseut »

dredmorbius wrote: post #97
zeidrich wrote: post #8

If you need them, Windows also has most of those tools somehow replicated in Powershell. For instance, the initial example can be replicated with: Get-Content .\data.csv | %{[int]$total+=$_.Split('|')[3]; } ; Write-Host "$total"

Or you can actually use the Linux commands by installing Cygwin. Pretty much my first conscious action when I wake up stranded on a desert Windows system.

Why Cygwin instead of MinGW? (I can't remember why I prefer MinGW, but at some point I had a reason).

Re: Useful Unix commands for data science

#100

Post by gwu78 »

lotsofcows wrote: post #94
gwu78 wrote: post #88

"Imagine you have a 4.2GB CSV file." " All you need... is the sum of all values in one particular column." In that case, if speed was paramount, I'd use Kona or kdb. Unquestionably, k is the best tool for that particular job.

Really? In a recent test of a whole bunch of languages, scripting, compiled and JVM (but not Kona or kdb), our awk test was beaten only by C. awk was so far ahead, its run time beat other's compile + run time.

Yes, really. Nothing I know of beats the speed of k for this column-oriented type of task. Kernighan himself tested it against awk many years ago and if I recall it was generally faster even in that set of tests.

Where can I see your experimental design? I'd like to try to replicate your results.

Post reply on HN