Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for exploring data

datavu.blogspot.com

Post reply on HN
Print view View original

141–150 of 156 posts

Re: Useful Unix commands for exploring data

#141

Post by collyw »

etrain wrote: post #127

Some more tips from someone who does this every day. 1) Be careful with CSV files and UNIX tools - most big CSV files with text fields have some subset of fields that are text quoted and character-escaped. This means that you might have "," in the middle of a string. Anything (like cut or awk) that depends on comma as a delimiter will not handle this situation well. 2) "cut" has shorter, easier to remember syntax tha…

Actually I would say Perl is more appropriate. I went back to Perl after 4 years for this sort of task, as it has so many features built into the syntax. Plus it can be run as a one liner.

Re: Useful Unix commands for exploring data

#142

Post by super_mario »

forkandwait wrote: post #131

"rs" for "reshape array". Found only on FreeBSD systems (yes, we are better... smile ) For example, transpose a text file: ~/ (j=0,r=1)$ cat foo.txt a b c d e f ~/ (j=0,r=0)$ cat foo.txt | rs -T a d b e c f Honestly I have never used in production, but I still think it is way cool. Also, being forced to work in a non-Unix environment, I am always reminded how much I wish everything were either text files, zipped text…

It's also on Mac OS X by default :D.

Re: Useful Unix commands for exploring data

#143

Post by collyw »

RBerenguel wrote: post #20
baldfat wrote: post #18

Certain people might miss the point of why to use command line. 1) I use this before using R or Python and ONLY do this when this is something I consistently need to be done all the time. Makes my R scripts shorter. 2) Somethings just need something simple to be fixed and these commands are just great. Learn awk and sed and your tools just go much larger in munging data.

Exactly! I had a longish period when I wanted to do everything with the same tool. Now, I try to pick the most efficient (for me, not the machine) to do it. Csvfix, awk, sed, jq and several other command line goodies make my life easier, the heavy lifting goes to R, gephi, or some ad-hoc Python, go or C

Fine if your only tool is Perl.

Re: Useful Unix commands for exploring data

#145

Post by mbreese »

vsbuffalo wrote: post #115
mbreese wrote: post #65

Earlier quoted context omitted.

It's not hard to write an on-disk merge sort using Python... it just may not be that fast. But really, as I'm sure you know, for genome-scale datasets, the key word is streaming. Disk IO is a major bottleneck. If you're using a large genomic dataset, you shouldn't be sorting your results in text format anyway... it would take way too much time and temporary disk space. What you'd probably want is a row filter to extr…

> If you're using a large genomic dataset, you shouldn't be sorting your results in text format anyway... it would take way too much time and temporary disk space. What you'd probably want is a row filter to extract out the rows of interest. For repeated queries, this isn't efficient. This is why we have indexed, sorted BAM files compressed with BGZF (and tabix, which uses the same ideas). Many queries in genomics ar…

I'm well aware of BAM and BGZF. I've even written a parser or two (yay double encoded binary formats). I really like the BGZF format and think it doesn't get used enough outside of sequencing. It's basically gzip frames with enough information to allow random access within the compressed stream. And tabix is a great way to efficiently index otherwise unindexable text files.

However, these are all binary formats. I specifically said that you shouldn't sort genome files in text format. Because while text is easy to parse, binary is faster for this. You aren't going to use any of the standard unix tools once you've crossed over into binary formats. And so you are stuck using domain specific tools. Stuff like http://ngsutils.org (shameless plug).

I have seen people write tools for parsing SAM files using unix core utils, but they are always orders of magnitude slower than a good BAM-specific tool (in almost any language).

Re: Useful Unix commands for exploring data

#146

Post by gaius »

aks_c wrote: post #117
zo1 wrote: post #100

Earlier quoted context omitted.

As someone that deals with large datasets on a Database + Python daily, I'm not quite sure what you mean. You'll have to explain it to me what "not a good idea is", or "basic data exploring".

Consider I get 10 files of size 3 GB every week, which I am supposed to filter based on certain column using a reference index and forward to my colleague. Before filtering I also want to check how the file looks like: column names, first few records etc. I can use something like following to explore few rows and few columns. $$ awk '{print $1,$3,$5}' file | head -10 And then I can use something like sed with referen…

For this kind of thing, it's easiest to bulk-load them into SQLite and do your exploration and early analysis in SQL

Re: Useful Unix commands for exploring data

#147

Post by rmc »

derekp7 wrote: post #105
epistasis wrote: post #38

Earlier quoted context omitted.

Yeah, I think calling him the "unenlightened" one is pretty off base here. For performing the tasks outlined by his examples, Unix utilities are easier for the user as well as executing faster than writing your own code in a general purpose programming language, unless one puts in the time to tune the implementation. One could rebuild AWK in C and get similar performance, but why not just use some extremely simple AW…

This is one area where I wish the Unix philosophy (reuse of tools) was taken a bit further. Too me, every command should be callable as a C library function. That way you wouldn't have to parse the human readable output through a pipe. Not only that, there needs to be both human-readable, as well as machine-readable output to all commands. For example I would love to be able to call "ps" from another script and easil…

But one of the unix philosophies is to use plain text. To have everything as a C function means everything needs a new API

Re: Useful Unix commands for exploring data

#148

Post by clarry »

mprovost wrote: post #93
bch wrote: post #79

Earlier quoted context omitted.

"Don't pipe a cat" is how I'm used to describing what you're talking about -- it may have been a performance issue in days past, but these days I think it's simply a matter of style. Not that style is not important.

This was drilled into me back in the usenet days. If you see a cat command with a single argument it's almost always replaceable by a shell redirection, or in this case just by passing the filename as an argument to grep. If you're processing lots of data like in the article there's no point in passing it through a separate command and pipe first.

But when you're interactively creating your pipeline, having the cat at the very beginning can save you some shuffling around. For example, while your second command is be grep and you're passing your file directly to grep, you might then realize you need another command before grep. So you'll have to move that argument to the other command. With a useless cat in front, you just insert the new command between cat and grep. It doesn't really cause harm.

Re: Useful Unix commands for exploring data

#149

Post by clarry »

jingo wrote: post #132
forkandwait wrote: post #131

"rs" for "reshape array". Found only on FreeBSD systems (yes, we are better... smile ) For example, transpose a text file: ~/ (j=0,r=1)$ cat foo.txt a b c d e f ~/ (j=0,r=0)$ cat foo.txt | rs -T a d b e c f Honestly I have never used in production, but I still think it is way cool. Also, being forced to work in a non-Unix environment, I am always reminded how much I wish everything were either text files, zipped text…

"Found only on FreeBSD..." Also found on NetBSD, OpenBSD and DragonFlyBSD.

Utree tells rs dates back to early 80s...

http://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/...

Re: Useful Unix commands for exploring data

#150

Post by zo1 »

Just one other thing I'd like to mention before everyone moves on to another topic. Not all of the unix commands are equal, and some have features that others don't.

E.g. I mainly work on AIX, and a lot of the commands are simply not the same as what they are on more standard linux flavors. From what I've heard, this applies between different distros as well.

Not so much the case with standard programming languages that are portable. E.g. Python. Unless you take in to account Jython, etc.

Post reply on HN