Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for exploring data

datavu.blogspot.com

Post reply on HN
Print view View original

131–140 of 156 posts

Re: Useful Unix commands for exploring data

#131

Post by forkandwait »

"rs" for "reshape array". Found only on FreeBSD systems (yes, we are better... smile)

For example, transpose a text file:

~/ (j=0,r=1)$ cat foo.txt a b c d e f ~/ (j=0,r=0)$ cat foo.txt | rs -T a d b e c f

Honestly I have never used in production, but I still think it is way cool.

Also, being forced to work in a non-Unix environment, I am always reminded how much I wish everything were either text files, zipped text files, or a SQL database. I know for really big data (bigger than our typical 10^7 row dataset, like imagery or genetics), you have to expand into things like HDF5, but part of my first data cleaning sequence is often to take something out of Excel or whatever and make a text file from it and apply unix tools.

Re: Useful Unix commands for exploring data

#132

Post by jingo »

forkandwait wrote: post #131

"rs" for "reshape array". Found only on FreeBSD systems (yes, we are better... smile ) For example, transpose a text file: ~/ (j=0,r=1)$ cat foo.txt a b c d e f ~/ (j=0,r=0)$ cat foo.txt | rs -T a d b e c f Honestly I have never used in production, but I still think it is way cool. Also, being forced to work in a non-Unix environment, I am always reminded how much I wish everything were either text files, zipped text…

"Found only on FreeBSD..."

Also found on NetBSD, OpenBSD and DragonFlyBSD.

Re: Useful Unix commands for exploring data

#133

Post by ars »

etrain wrote: post #127

Some more tips from someone who does this every day. 1) Be careful with CSV files and UNIX tools - most big CSV files with text fields have some subset of fields that are text quoted and character-escaped. This means that you might have "," in the middle of a string. Anything (like cut or awk) that depends on comma as a delimiter will not handle this situation well. 2) "cut" has shorter, easier to remember syntax tha…

> you almost invevitably want to run sort before you run uniq

And then you don't actually want uniq anyway since sort has a -u switch that removes duplicate lines.

Re: Useful Unix commands for exploring data

#135

Post by tstack »

tdicola wrote: post #69

If you're on Windows, you owe it to yourself to check out a little known Microsoft utility called logparser: http://mlichtenberg.wordpress.com/2011/02/03/log-parser-rock... It effectively lets you query a CSV (or many other log file formats/sources) with a SQL-like language. Very useful tool that I wish was available on Linux systems.

lnav (http://lnav.org) provides SQL-queries-over-logs in the unix world. It's also a nice viewer for the logs themselves.

Re: Useful Unix commands for exploring data

#136

Post by potatosareok »

ars wrote: post #133
etrain wrote: post #127

Some more tips from someone who does this every day. 1) Be careful with CSV files and UNIX tools - most big CSV files with text fields have some subset of fields that are text quoted and character-escaped. This means that you might have "," in the middle of a string. Anything (like cut or awk) that depends on comma as a delimiter will not handle this situation well. 2) "cut" has shorter, easier to remember syntax tha…

> you almost invevitably want to run sort before you run uniq And then you don't actually want uniq anyway since sort has a -u switch that removes duplicate lines.

What if you want uniq -c? Any simple way to replicate that functionality better then...sort | uniq -c?

Re: Useful Unix commands for exploring data

#137

Post by ars »

potatosareok wrote: post #136
ars wrote: post #133

Earlier quoted context omitted.

> you almost invevitably want to run sort before you run uniq And then you don't actually want uniq anyway since sort has a -u switch that removes duplicate lines.

What if you want uniq -c? Any simple way to replicate that functionality better then...sort | uniq -c?

Then you run uniq -c (which I do all the time).

But for the examples in the main article sort -u would be fine.

Re: Useful Unix commands for exploring data

#138

Post by barrkel »

riffraff wrote: post #119
barrkel wrote: post #84

Earlier quoted context omitted.

The problem is that the most obvious way of doing it - File.readlines('foo.txt').map { ... }.select { ... } etc. - is not stream-oriented.

arguably, it's trivial to make that stream oriented open('tmp.rb').each_line.lazy.map {...}.select {...} the problem with processing big files with ruby (in my humble experience) is usually that it's still slow enough that "preprocessing with grep&uniq" is worthwhile.

    > open('tmp.rb').each_line.lazy
    NoMethodError: undefined method `lazy' for #:each_line>
Not everybody is using Ruby 2.0.

Re: Useful Unix commands for exploring data

#139

Post by unhammer »

andreasvc wrote: post #107
unhammer wrote: post #96

Earlier quoted context omitted.

ag is often faster when you're using it interactively, replacing "grep -r" (in particular in version controlled dirs). It's also faster in the sense that for interactive use it will often DWYM. But has too many weird quirks that it can replace grep for data munging. E.g. $ ag verb fullform_nn.txt >/tmp/verbs ERR: Too many matches in fullform_nn.txt. Skipping the rest of this file. Man ag says there's a --max-count op…

Did you report it as a bug?

did now https://github.com/ggreer/the_silver_searcher/issues/483 (though it took me a while to figure out a more precise issue title than "--max-count is taunting me")

Re: Useful Unix commands for exploring data

#140

Post by unhammer »

nailer wrote: post #15

I love Unix pipelines, but chances are your data is structured in such a way that using regex based tools will break that structure unless you're very careful. You know that thing about not making HTML with regexs? Same rule applies to CSV, TSV, and XLSX. All these can be created, manipulated and read using Python, which is probably already on your system.

There are unix tools that handle XML or CVS as well ;-) http://wiki.apertium.org/wiki/Xml_grep http://csvkit.readthedocs.org/en/latest/
Post reply on HN