Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for exploring data

datavu.blogspot.com

Post reply on HN
Print view View original

91–100 of 156 posts

Re: Useful Unix commands for exploring data

#91

Post by dbro »

napD wrote: post #6
sheetjs wrote: post #2

caveat: delimiter-based commands are not quote-aware. For example, this is a CSV line with two fields: foo,"bar,baz" However, the tools will treat it as 3 columns: $ echo 'foo,"bar,baz"' | awk -F, '{print NF}' 3

Is there any workaround?

yes!

https://github.com/dbro/csvquote

csvquote allows UNIX tools to work properly with quoted fields that contain delimiters inside the data. It is a simple translation tool that temporarily replaces the special characters occurring inside quotes with harmless non-printing characters. You do it as a first step in the pipeline, then do the regular operations using UNIX tools, and the last step of of the pipeline restores those troublesome characters back inside the data fields.

Re: Useful Unix commands for exploring data

#92

Post by discardorama »

danielweber wrote: post #90
aks_c wrote: post #10

Earlier quoted context omitted.

Thank you for inputs, how about this? uniq -u movies.csv > temp.csv temp.csv > movie.csv rm temp.csv

long_running_process > filename.tmp && mv filename.tmp filename The rename is atomic; anyone opening "filename" will get either the old version, or the new version. (Although it breaks one of my other favorite idioms for monitoring log files, "tail -f filename", because the old inode will never be updated.)

> Although it breaks one of my other favorite idioms for monitoring log files, "tail -f filename", because the old inode will never be updated

You should look into the '-F' option of tail; it follows the filename, and not the inode.

Re: Useful Unix commands for exploring data

#93

Post by mprovost »

bch wrote: post #79
mprovost wrote: post #63

Earlier quoted context omitted.

Useless use of cat?

"Don't pipe a cat" is how I'm used to describing what you're talking about -- it may have been a performance issue in days past, but these days I think it's simply a matter of style. Not that style is not important.

This was drilled into me back in the usenet days. If you see a cat command with a single argument it's almost always replaceable by a shell redirection, or in this case just by passing the filename as an argument to grep. If you're processing lots of data like in the article there's no point in passing it through a separate command and pipe first.

Re: Useful Unix commands for exploring data

#95

Post by 727374 »

tdicola wrote: post #69

If you're on Windows, you owe it to yourself to check out a little known Microsoft utility called logparser: http://mlichtenberg.wordpress.com/2011/02/03/log-parser-rock... It effectively lets you query a CSV (or many other log file formats/sources) with a SQL-like language. Very useful tool that I wish was available on Linux systems.

LogParser is one of the few things I really miss from windows. I think there are unix equivalents, but I haven't had the time to invest in learning them. Pretty much every example in this article boiled down to 'Take this CSV and run a simple SQL query on it'. Yes you can do that by piping through various unix utilities or you could just use a tool mean specifically for the task. I'd like to see the article explore some more advanced cases, like rolling up a column. I actually had to do this yesterday and ended up opening my data in open office and using a pivot table.

Re: Useful Unix commands for exploring data

#96

Post by unhammer »

namplaa wrote: post #86
georgegeorges wrote: post #68

Earlier quoted context omitted.

> always use the right tool for the job Standard grep is much faster on multi-gigabyte files than anything you can figure out how to do in your pet language. By the time you get close to matching grep, you would have reimplemented most of grep, in half-assed fashion at that. Your delusion is assuming standard command line tools are simple in function because they have a simple interface that Average Joe can use.

Personally I find silversearcher (ag) faster, and reinventing standard commandline tools with a collection of other tools is often slower. One liner shell commands often turn complicated quickly.

ag is often faster when you're using it interactively, replacing "grep -r" (in particular in version controlled dirs). It's also faster in the sense that for interactive use it will often DWYM.

But has too many weird quirks that it can replace grep for data munging. E.g.

    $ ag verb fullform_nn.txt >/tmp/verbs
    ERR: Too many matches in fullform_nn.txt. Skipping the rest of this file.
Man ag says there's a --max-count option. Let's try that.

    $ grep -c verb fullform_nn.txt
    206077
    $ ag --max-count 206077 verb fullform_nn.txt >/tmp/verbs
    ERR: Too many matches in fullform_nn.txt. Skipping the rest of this file.
Wtf? (and running those two commands with "time" gave ag user 0m0.770s while grep had user 0m0.057s)

Re: Useful Unix commands for exploring data

#100

Post by zo1 »

aks_c wrote: post #80
zo1 wrote: post #32

"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how yo…

When you have to deal with (genetic data) files of few GB on daily basis, I dont think using Python, R or databases is good idea to do basic data exploring. -rwxr-x--- 1 29594528008 out_chr1comb.dose -rwxr-x--- 1 27924241334 out_chr2comb.dose -rwxr-x--- 1 25684164559 out_chr3comb.dose -rwxr-x--- 1 24665680612 out_chr4comb.dose -rwxr-x--- 1 21493584686 out_chr5comb.dose -rwxr-x--- 1 23626967979 out_chr6comb.dose -rwxr…

As someone that deals with large datasets on a Database + Python daily, I'm not quite sure what you mean. You'll have to explain it to me what "not a good idea is", or "basic data exploring".
Post reply on HN