Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for exploring data

datavu.blogspot.com

Post reply on HN
Print view View original

81–90 of 156 posts

Re: Useful Unix commands for exploring data

#81

Post by cyphunk »

really HN? if you find yourself depending heavily on the recommendations in this article you are doing data analysis wrong. Shell foo is relevant to data analysis only as much as regex is. In the same light depending on these methods too much is digging a deep knowledge ditch that in the end is going to limit and hinder you way more than the initial ingress time required to learn more capable data analytics frameworks or at least a scripting language.

still, on international man page appreciate day this is a great reference. the only thing it is missing is gnuplot ascii graphs.

Re: Useful Unix commands for exploring data

#82

Post by Terretta »

dpeck wrote: post #62
barrkel wrote: post #39

Earlier quoted context omitted.

Most scripting languages aren't multithreaded, and some aren't pipeline oriented by default. For example, working with file lines naively in Ruby means reading the whole lot into a giant array and doing transformations an array at a time, rather than in a streaming fashion. The shell gives you fairly safe concurrency and streaming for free. Personally, if it's a complex task, I generally write a tool such that it can…

No the case with ruby at all, if you're reading the whole file into memory theres a good chance you're doing it wrong. check out yield and blocks

> if you're reading the whole file into memory theres a good chance you're doing it wrong

GP: "working with file lines _naively_ in Ruby"

Re: Useful Unix commands for exploring data

#84

Post by barrkel »

dpeck wrote: post #62
barrkel wrote: post #39

Earlier quoted context omitted.

Most scripting languages aren't multithreaded, and some aren't pipeline oriented by default. For example, working with file lines naively in Ruby means reading the whole lot into a giant array and doing transformations an array at a time, rather than in a streaming fashion. The shell gives you fairly safe concurrency and streaming for free. Personally, if it's a complex task, I generally write a tool such that it can…

No the case with ruby at all, if you're reading the whole file into memory theres a good chance you're doing it wrong. check out yield and blocks

The problem is that the most obvious way of doing it - File.readlines('foo.txt').map { ... }.select { ... } etc. - is not stream-oriented.

Re: Useful Unix commands for exploring data

#86

Post by namplaa »

georgegeorges wrote: post #68
zo1 wrote: post #32

"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how yo…

> always use the right tool for the job Standard grep is much faster on multi-gigabyte files than anything you can figure out how to do in your pet language. By the time you get close to matching grep, you would have reimplemented most of grep, in half-assed fashion at that. Your delusion is assuming standard command line tools are simple in function because they have a simple interface that Average Joe can use.

Personally I find silversearcher (ag) faster, and reinventing standard commandline tools with a collection of other tools is often slower.

One liner shell commands often turn complicated quickly.

Re: Useful Unix commands for exploring data

#87

Post by namplaa »

CraigJPerry wrote: post #3

>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There…

You might want to use sort -u

Re: Useful Unix commands for exploring data

#89

Post by teh_klev »

tdicola wrote: post #69

If you're on Windows, you owe it to yourself to check out a little known Microsoft utility called logparser: http://mlichtenberg.wordpress.com/2011/02/03/log-parser-rock... It effectively lets you query a CSV (or many other log file formats/sources) with a SQL-like language. Very useful tool that I wish was available on Linux systems.

LogParser is excellent, so many Windows admins have never heard of it which is a shame.

Re: Useful Unix commands for exploring data

#90

Post by danielweber »

aks_c wrote: post #10
CraigJPerry wrote: post #3

>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There…

Thank you for inputs, how about this? uniq -u movies.csv > temp.csv temp.csv > movie.csv rm temp.csv

long_running_process > filename.tmp && mv filename.tmp filename

The rename is atomic; anyone opening "filename" will get either the old version, or the new version. (Although it breaks one of my other favorite idioms for monitoring log files, "tail -f filename", because the old inode will never be updated.)

Post reply on HN