Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for data science

gregreda.com

Post reply on HN
Print view View original

51–60 of 108 posts

Re: Useful Unix commands for data science

#51

Post by jbert »

chubot wrote: post #27
jbert wrote: post #3

I like slicing and dicing with awk, grep and friends too. One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility? echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command I suppose more generally you could have a 'fold initial op' and: echo "1\n2\n3\n4\n" | fold 0 + # should print 10 echo "1\n2\n3\n4\n" | fold 1 \* # should print…

Yeah I wrote my own sum utility in Python... the syntax is just sum 1 or sum 2 for the column, with a -d delimiter flag. In retrospect I guess it could have been a one line awk script. But yeah if you are doing this kind of data-processing, it makes sense to have a hg/git repo of aliases and tiny commands that you sync around from machine to machine. You shouldn't have to write the sum more than once. Another useful…

I've never aliased it, but yes I use your 'hist' a lot. Useful for things like "categorise log errors" etc.

Does everyone else edit command history, stacking up 'grep -v xxxx' in the pipeline to remove noise?

If I'm working on a new pipeline, my normal workflow is something like:

  head file     # See some representative lines
  head file | grep goodstuff
  head file | grep good stuff | grep -v badstuff
  head file | grep ... | grep ... | sed -e 's/cut out/bits/' -e 's/i dont/want/'
  head file | grep ... | grep ... | sed -e 's/cut out/bits/' -e 's/i dont/want/' | awk '{print $3}' # get a col
  head file | grep ... | grep ... | sed -e 's/cut out/bits/' -e 's/i dont/want/' | awk '{print $3}' | sort | uniq -c | sort -nr  # histogram as parent
Then I edit the 'head' into a 'cat' and handle the whole file. Basically all done with bash history editing (I'm a 'set -o vi' person for vi keybindings in bash, emacs is fine too :-)

Re: Useful Unix commands for data science

#52

Post by nnnnni »

aidos wrote: post #47
bch wrote: post #45

> cat data.csv | awk -F "|" '{ sum += $4 } END { printf "%.2f\n", sum }' "Don't pipe a cat". My test doesn't show a speed improvement, but there are less processes running, and less memory consumed. bch:~ bch$ jot 999999999 2 99 > data.dat bch:~ bch$ time cat data.dat | awk '{sum +=$1} END {printf "sum: %d\n", sum}' sum: 50499999412 real 6m21.111s user 6m15.506s sys 0m5.711s PID COMMAND %CPU TIME #TH #WQ #PORTS #MREG…

Sometimes I like to start with cat so I can easily swap for zcat when changing to gripped input.

Hmm… Now we need a zawk.

Re: Useful Unix commands for data science

#53

Post by jbert »

aidos wrote: post #47
bch wrote: post #45

> cat data.csv | awk -F "|" '{ sum += $4 } END { printf "%.2f\n", sum }' "Don't pipe a cat". My test doesn't show a speed improvement, but there are less processes running, and less memory consumed. bch:~ bch$ jot 999999999 2 99 > data.dat bch:~ bch$ time cat data.dat | awk '{sum +=$1} END {printf "sum: %d\n", sum}' sum: 50499999412 real 6m21.111s user 6m15.506s sys 0m5.711s PID COMMAND %CPU TIME #TH #WQ #PORTS #MREG…

Sometimes I like to start with cat so I can easily swap for zcat when changing to gripped input.

Agree. Or actually I start with a 'head -100' so I don't handle too much data in my pipeline until it's ready.

Re: Useful Unix commands for data science

#54

Post by McUsr »

foobarbazqux wrote: post #31
daemon13 wrote: post #17

Earlier quoted context omitted.

I recall there was a pointer to an old great AWK tutorial some time ago - smth along the lines 'how to approach awk language....' - anyone kept the link?

This is the first hit for awk tutorial and it's all you need. http://www.grymoire.com/Unix/Awk.html

I think Steve's Awk academy is a nice supplement to Grymoire : http://www.troubleshooters.com/codecorn/awk/

By the way: what people need to understand is that in order to use Awk, efficently, you'll either use associative arrays, or structure your script like a sed script, otherwise it will be slow. The interesting thing about both of those, is the regex algorithm Thompson NFA, that is from what I hear around 7 times faster than PCRE that is used in Perl, PHP, Python and Ruby?

Re: Useful Unix commands for data science

#55

Post by mjn »

jbert wrote: post #51
chubot wrote: post #27

Earlier quoted context omitted.

Yeah I wrote my own sum utility in Python... the syntax is just sum 1 or sum 2 for the column, with a -d delimiter flag. In retrospect I guess it could have been a one line awk script. But yeah if you are doing this kind of data-processing, it makes sense to have a hg/git repo of aliases and tiny commands that you sync around from machine to machine. You shouldn't have to write the sum more than once. Another useful…

I've never aliased it, but yes I use your 'hist' a lot. Useful for things like "categorise log errors" etc. Does everyone else edit command history, stacking up 'grep -v xxxx' in the pipeline to remove noise? If I'm working on a new pipeline, my normal workflow is something like: head file # See some representative lines head file | grep goodstuff head file | grep good stuff | grep -v badstuff head file | grep ... |…

Yeah, this is my quick-and-dirty way of looking at referers in Apache logs, built up from a few history edits. It excludes some bot-like stuff (many bots give a plus-prefixed URL in the user-agent string) and referer strings from my own domain, removes query strings, and cleans up trailing slashes:

   grep -v "+http" access_log | cut -d \" -f 4 | cut -d \? -f 1 | sed 's/\/$//' | grep -v kmjn.org | sort | uniq -c | sort -nr

Re: Useful Unix commands for data science

#56

Post by mcavage »

jlgaddis wrote: post #40
res0nat0r wrote: post #7

If this interests you, you should check out Joyents new Manta service which lets you do this type of thing on your data via their infrastructure. It's really cool. http://www.joyent.com/products/manta

If I needed to do this type of thing on 10 TB of data, it would probably take me longer to get the data to them than it would to just run it on my own hardware. Apparently there's a need for it, though, or it wouldn't exist.

Disclaimer: I work at Joyent, on Manta.

This entire HN thread is a perfect example of why we built Manta. Lots of engineers/scientists/sysadmins/... already know how to (elegantly) process data using Unix and augmenting with scripts. Manta isn't about always needing to work on a 10TB dataset (you can), but about it being always available, and stored ready to go. I know we can't live without it for running our own systems -- all logs in the entire Joyent fleet are rotated and archived in Manta, and we can perform both recurring/automated and ad-hoc analysis on the dataset, without worrying about storage shares, or ETL'ing from cold storage to compute, etc. And you can sample as little as much or as much as you want. At least to us (and I've run several large distributed systems in my career), that has tremendous value, and we believe it does for others as well. And that's just one use case (log processing).

Like I said, disclaimers/bias/etc.

m

Re: Useful Unix commands for data science

#57

Post by thrownaway2424 »

dsberkholz wrote: post #43
foobarbazqux wrote: post #24

Earlier quoted context omitted.

Gnu grep is or was very slow with the UTF-8 locale. Not sure about other commands, perhaps anything that processes text, awk and sed maybe?

That was mostly fixed. http://savannah.gnu.org/bugs/?14472

It's no longer quadratic in so many cases, but it's still true that UTF-8 string operations require, in the best case, several CPU cycles per character consumed, even when the input is an ASCII subset. LC_ALL=C pretty much guarantees one or fewer CPU cycles per input character. Basics like strlen and strchr and strstr are significantly faster in "C" locale.

Re: Useful Unix commands for data science

#60

Post by DEinspanjer »

One of my favorite little tools that makes all these others better is pv -- PipeViewer.

Use it any place in a pipeline to see a progress meter on stderr. Very handy when grepping through a bunch of big log files looking for stuff. Here is a quick strawman example:

  pv /data/*.log.gz | zgrep -c 'hello world'
  241MiB 0:00:15 [15.8MiB/s] [==>       ]  2% ETA 0:12:12
Post reply on HN