Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for exploring data

datavu.blogspot.com

Post reply on HN
Print view View original

71–80 of 156 posts

Re: Useful Unix commands for exploring data

#71

Post by gopalv »

zo1 wrote: post #32

"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how yo…

I would allow that for some definition of "too-big".

I wrote a distributed grep impl a few years back to grep my logs and collect output to a central machine (a vague "how may machines had this error" job).

The central orchestration was easy in python, but implementing

zgrep | awk | sort | uniq -c | wc -l

is way faster and way more code in python than to do it with shell (zgrep is awesome for .gz logs).

On the other hand, the shell co-ordinator way way harder using pdsh that I reverted to using paramiko and python threadpools.

Unix tools are extremely composable and present in nearly every machine with the standard behaviour.

Re: Useful Unix commands for exploring data

#72

Post by jason_slack »

coderholic wrote: post #57
jason_slack wrote: post #43

The author states: uniq -u movies.csv > temp.csv mv temp.csv movie.csv **Important thing to note here is uniq wont work if duplicate records are not adjacent. [Addition based on HN inputs] Would the fix here be to sort the lines first using the `sort` command first? Then `uniq`?

Yes, and in fact you can just use sort's -u (unqiue) argument, and avoid uniq all together.

Thanks for the clarification!

Re: Useful Unix commands for exploring data

#73

Post by jason_slack »

icebraining wrote: post #56
jason_slack wrote: post #43

The author states: uniq -u movies.csv > temp.csv mv temp.csv movie.csv **Important thing to note here is uniq wont work if duplicate records are not adjacent. [Addition based on HN inputs] Would the fix here be to sort the lines first using the `sort` command first? Then `uniq`?

Yes, but not first, rather instead. "sort -u" both sorts and hides duplicates.

Thanks for the clarification!

Re: Useful Unix commands for exploring data

#76

Post by oftenwrong »

rmc wrote: post #26
daurnimator wrote: post #22

Earlier quoted context omitted.

The utility to do this is called sponge. http://linux.die.net/man/1/sponge uniq -u movies.csv | sponge movies.csv

sponge is cool. But on debian/ubuntu, it's packaged up in moreutils, which includes a few helpful tools. However a programme called parallel is in moreutils, and that's not as powerful as GNU's parallel. So I often end up uninstalling sponge/moreutils. :(

GNU Parallel is an indispensable heavy lifter on the command line. I was expecting it to show up in the article.

Re: Useful Unix commands for exploring data

#77

Post by jeroenjanssens »

zerop wrote: post #48

7 command-line tools for data science http://jeroenjanssens.com/2013/09/19/seven-command-line-tool... Useful Unix commands for data science http://www.gregreda.com/2013/07/15/unix-commands-for-data-sc...

First blog post was the inspiration for a book, which is almost finished: http://datascienceatthecommandline.com

Re: Useful Unix commands for exploring data

#79

Post by bch »

mprovost wrote: post #63
LiveTheDream wrote: post #61

I use this command very frequently to check how often an event occurs in a log file over time (specifically in 10-minute buckets), assuming the file is formatting like "INFO - [2014-08-27 16:16:29,578] Something something something" cat /path/to/logfile | grep PATTERN | sed 's/.*\(2014-..-..\) \(..\):\(.\).*/\1 \2:\3x/' | uniq -c results in: 273 2014-08-27 14:5x 222 2014-08-27 15:0x 201 2014-08-27 15:1x 171 2014-08-2…

Useless use of cat?

"Don't pipe a cat" is how I'm used to describing what you're talking about -- it may have been a performance issue in days past, but these days I think it's simply a matter of style. Not that style is not important.

Re: Useful Unix commands for exploring data

#80

Post by aks_c »

zo1 wrote: post #32

"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how yo…

When you have to deal with (genetic data) files of few GB on daily basis, I dont think using Python, R or databases is good idea to do basic data exploring.

-rwxr-x--- 1 29594528008 out_chr1comb.dose

-rwxr-x--- 1 27924241334 out_chr2comb.dose

-rwxr-x--- 1 25684164559 out_chr3comb.dose

-rwxr-x--- 1 24665680612 out_chr4comb.dose

-rwxr-x--- 1 21493584686 out_chr5comb.dose

-rwxr-x--- 1 23626967979 out_chr6comb.dose

-rwxr-x--- 1 20856136599 out_chr7comb.dose

-rwxr-x--- 1 18398180426 out_chr8comb.dose

-rwxr-x--- 1 15864714472 out_chr9comb.dose

Post reply on HN