Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for exploring data

datavu.blogspot.com

Post reply on HN
Print view View original

111–120 of 156 posts

Re: Useful Unix commands for exploring data

#111

Post by vdm »

CGamesPlay wrote: post #42

For working with complex CSV files, I highly recommend checking out CSVKit https://csvkit.readthedocs.org/en/0.8.0/ I've just started using it, and the only limitation I've so far encountered has been that there's no equivalent to awk (i.e. I want a way to evaluate a python expression on every line as part of a pipeline).

Get words starting with "and"

    $ cat /usr/share/dict/words | py -fx 're.match(r"and", x)' | head -5
    and
    andante
    andante's
    andantes
    andiron
https://github.com/Russell91/pythonpy

Re: Useful Unix commands for exploring data

#112

Post by vdm »

tdicola wrote: post #69

If you're on Windows, you owe it to yourself to check out a little known Microsoft utility called logparser: http://mlichtenberg.wordpress.com/2011/02/03/log-parser-rock... It effectively lets you query a CSV (or many other log file formats/sources) with a SQL-like language. Very useful tool that I wish was available on Linux systems.

Nice find. You might also like http://matt.might.net/articles/sql-in-the-shell/

Re: Useful Unix commands for exploring data

#114

Post by sanderjd »

derekp7 wrote: post #105
epistasis wrote: post #38

Earlier quoted context omitted.

Yeah, I think calling him the "unenlightened" one is pretty off base here. For performing the tasks outlined by his examples, Unix utilities are easier for the user as well as executing faster than writing your own code in a general purpose programming language, unless one puts in the time to tune the implementation. One could rebuild AWK in C and get similar performance, but why not just use some extremely simple AW…

This is one area where I wish the Unix philosophy (reuse of tools) was taken a bit further. Too me, every command should be callable as a C library function. That way you wouldn't have to parse the human readable output through a pipe. Not only that, there needs to be both human-readable, as well as machine-readable output to all commands. For example I would love to be able to call "ps" from another script and easil…

PowerShell solves this problem by piping around objects instead of strings. It's pretty neat!

Re: Useful Unix commands for exploring data

#115

Post by vsbuffalo »

mbreese wrote: post #65
vsbuffalo wrote: post #50

Earlier quoted context omitted.

> Hate to sound like Steve-Jobs here, but: "You're using it wrong." I don't quite agree — say this individual needs to sort a file by two columns. Should they really load everything into memory to call Python's sorted()? With large genomics datasets this isn't possible. Trying to reimplement sort's on-disk merge sort would be unnecessary and treacherous. It's easy to forget how much engineer went into these core util…

It's not hard to write an on-disk merge sort using Python... it just may not be that fast. But really, as I'm sure you know, for genome-scale datasets, the key word is streaming. Disk IO is a major bottleneck. If you're using a large genomic dataset, you shouldn't be sorting your results in text format anyway... it would take way too much time and temporary disk space. What you'd probably want is a row filter to extr…

> If you're using a large genomic dataset, you shouldn't be sorting your results in text format anyway... it would take way too much time and temporary disk space. What you'd probably want is a row filter to extract out the rows of interest.

For repeated queries, this isn't efficient. This is why we have indexed, sorted BAM files compressed with BGZF (and tabix, which uses the same ideas). Many queries in genomics are with respect to position, and this is O(1) with a sorted, indexed file and O(n) with streaming. Streaming also involves reading and uncompressing the entire file from the disk — accessing entries with from an indexed, sorted file involves a seek() to the correct offset and decompressing that particular block – this is far more efficient. I definitely agree that streaming is great, but sorting data is an essential trick in working with large genomics data sets.

Re: Useful Unix commands for exploring data

#116

Post by prlin »

WestCoastJustin wrote: post #53

My personal favorite is to use this pattern. You can do some extremely cool counts and group by operations at the command like [1]: grep '01/Jul/1995' NASA_access_log_Jul95 | awk '{print $1}' | sort | uniq -c | sort -h -r | head -n 15 Turns this: 199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245 unicomp6.unicomp.net - - [01/Jul/1995:00:00:06 -0400] "GET /shuttle/countdown/ HTTP/1.…

Similar ideas in Python with Generators and Co-routines:

http://www.dabeaz.com/generators-uk/

Re: Useful Unix commands for exploring data

#117

Post by aks_c »

zo1 wrote: post #100
aks_c wrote: post #80

Earlier quoted context omitted.

When you have to deal with (genetic data) files of few GB on daily basis, I dont think using Python, R or databases is good idea to do basic data exploring. -rwxr-x--- 1 29594528008 out_chr1comb.dose -rwxr-x--- 1 27924241334 out_chr2comb.dose -rwxr-x--- 1 25684164559 out_chr3comb.dose -rwxr-x--- 1 24665680612 out_chr4comb.dose -rwxr-x--- 1 21493584686 out_chr5comb.dose -rwxr-x--- 1 23626967979 out_chr6comb.dose -rwxr…

As someone that deals with large datasets on a Database + Python daily, I'm not quite sure what you mean. You'll have to explain it to me what "not a good idea is", or "basic data exploring".

Consider I get 10 files of size 3 GB every week, which I am supposed to filter based on certain column using a reference index and forward to my colleague. Before filtering I also want to check how the file looks like: column names, first few records etc.

I can use something like following to explore few rows and few columns. $$ awk '{print $1,$3,$5}' file | head -10

And then I can use something like sed with reference index to filter the file. Since I plan to repeat this with different files, databases would be time consuming(even if I automate it loading every file and querying). Due to the file size options like R, Python would be slower than unix commands. I can also save set of commands as script and share/run whenever I need it.

If there is a better way I would be happy to learn.

Re: Useful Unix commands for exploring data

#118

Post by xorcist »

namplaa wrote: post #86
georgegeorges wrote: post #68

Earlier quoted context omitted.

> always use the right tool for the job Standard grep is much faster on multi-gigabyte files than anything you can figure out how to do in your pet language. By the time you get close to matching grep, you would have reimplemented most of grep, in half-assed fashion at that. Your delusion is assuming standard command line tools are simple in function because they have a simple interface that Average Joe can use.

Personally I find silversearcher (ag) faster, and reinventing standard commandline tools with a collection of other tools is often slower. One liner shell commands often turn complicated quickly.

I have never used ag, but in most instances where people thought they made a faster grep it is becuase it doesn't handle multibyte encodings correctly.

Have you rerun your tests in the "C" locale?

Re: Useful Unix commands for exploring data

#119

Post by riffraff »

barrkel wrote: post #84
dpeck wrote: post #62

Earlier quoted context omitted.

No the case with ruby at all, if you're reading the whole file into memory theres a good chance you're doing it wrong. check out yield and blocks

The problem is that the most obvious way of doing it - File.readlines('foo.txt').map { ... }.select { ... } etc. - is not stream-oriented.

arguably, it's trivial to make that stream oriented

    open('tmp.rb').each_line.lazy.map {...}.select {...}
the problem with processing big files with ruby (in my humble experience) is usually that it's still slow enough that "preprocessing with grep&uniq" is worthwhile.

Re: Useful Unix commands for exploring data

#120

Post by jbooth »

aks_c wrote: post #117
zo1 wrote: post #100

Earlier quoted context omitted.

As someone that deals with large datasets on a Database + Python daily, I'm not quite sure what you mean. You'll have to explain it to me what "not a good idea is", or "basic data exploring".

Consider I get 10 files of size 3 GB every week, which I am supposed to filter based on certain column using a reference index and forward to my colleague. Before filtering I also want to check how the file looks like: column names, first few records etc. I can use something like following to explore few rows and few columns. $$ awk '{print $1,$3,$5}' file | head -10 And then I can use something like sed with referen…

I think the gain you're seeing there is because it's quicker for you to do quick, dirty ad hoc work with the shell than it is to write custom python for each file. Which totally makes sense, the work's ad hoc so use an ad hoc tool. Python being slow and grep being a marvel of optimization doesn't really matter, here, compared to the dev time you're saving.
Post reply on HN