Using AWK and R to parse 25TB • Hacker News Board

Re: Using AWK and R to parse 25TB

#11

Post by stirfrykitty » Thu, Jun 27, 2019, 1:42 PM UTC

There was a similar article (2014) that is also interesting. I think too many of us see new and shiny and immediately glom onto it, forgetting that the UNIX/regex fathers knew a thing or two about crunching data.

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

Re: Using AWK and R to parse 25TB

#12

Post by kohtatsu » Thu, Jun 27, 2019, 1:43 PM UTC

markus_zhang wrote: post #5

innomatics wrote: post #4

Great post and thanks for sharing your learnings. A couple of quick questions: Was the 25TB raw data gathered from a single human genome? What would be the size in bytes of a unique genomic fingerprint once raw data is all fully processed into high confidence base values? (including non-coding regions) If we just look at coding regions and further compress by only looking at SNPs, how many bytes is that? Considering…

Sorry I saw this article and thought it was pretty interesting. This is NOT my article but I'd like to know what others would do under this situation. BTW not sure, but is it OK to post other's article here? Maybe I should add a short commentary in the title.

The default assumption is that it's not your article, unless you prepend "Show HN" (or there's something obvious like your username matching the domain name).

Re: Using AWK and R to parse 25TB

#13

Post by rout39574 » Thu, Jun 27, 2019, 1:52 PM UTC

stirfrykitty wrote: post #11

There was a similar article (2014) that is also interesting. I think too many of us see new and shiny and immediately glom onto it, forgetting that the UNIX/regex fathers knew a thing or two about crunching data. https://adamdrake.com/command-line-tools-can-be-235x-faster-...

You might be on to something there.

Re: Using AWK and R to parse 25TB

#14

Post by close04 » Thu, Jun 27, 2019, 1:54 PM UTC

stirfrykitty wrote: post #11

There was a similar article (2014) that is also interesting. I think too many of us see new and shiny and immediately glom onto it, forgetting that the UNIX/regex fathers knew a thing or two about crunching data. https://adamdrake.com/command-line-tools-can-be-235x-faster-...

> often people use Hadoop and other so-called Big Data ™ tools for real-world processing and analysis jobs that can be done faster with simpler tools and different techniques.

Right tool for the right job, as always. For a 2-3GB dataset size you don't need to bother with Hadoop just as for a 2-3PB dataset size you probably don't need to bother with awk.

Re: Using AWK and R to parse 25TB

#15

Post by sansnomme » Thu, Jun 27, 2019, 1:55 PM UTC

What's the best pipeline/workflow management tool for command line programs with a GUI? E.g. for resuming the process after it gets interrupted etc.

Re: Using AWK and R to parse 25TB

#16

Post by et2o » Thu, Jun 27, 2019, 1:56 PM UTC

My biggest question is why on earth did they give you 25 Tb of TSV genetic data? :-)

I'm not sure what your sample was but seems like it would have been better to use one of the special binary file formats for genetic data. You wrote SNP chips, But in order to get to 25 Tb I assume there must be imputed calls, so it seems like a BGEN might have been a lot easier.

This is speculation of course, I'm not sure exactly what your situation was.

Re: Using AWK and R to parse 25TB

#17

Post by c0l0 » Thu, Jun 27, 2019, 2:02 PM UTC

sansnomme wrote: post #15

What's the best pipeline/workflow management tool for command line programs with a GUI? E.g. for resuming the process after it gets interrupted etc.

I am not kidding: A graphical terminal emulator.

Mastering the usual command line interface (terminal emulator, interactive shell, maybe a terminal multiplexer) is non-optional if you want to use CLI tools at or close to peak effectiveness.

Re: Using AWK and R to parse 25TB

#18

Post by sansnomme » Thu, Jun 27, 2019, 2:05 PM UTC

c0l0 wrote: post #17

sansnomme wrote: post #15

What's the best pipeline/workflow management tool for command line programs with a GUI? E.g. for resuming the process after it gets interrupted etc.

I am not kidding: A graphical terminal emulator. Mastering the usual command line interface (terminal emulator, interactive shell, maybe a terminal multiplexer) is non-optional if you want to use CLI tools at or close to peak effectiveness.

I mean task resumption after interruption etc. Like airflow type of tools. Not quite unix task suspend options, this is about data pipelines. For Hadoop-style MapReduce, you can split the task into jobs which can be resumed and discarded etc. Shell scripting is not an elegant way to deal with this, a proper orchestrator tool is better.

Re: Using AWK and R to parse 25TB

#19

Post by scottlocklin » Thu, Jun 27, 2019, 2:08 PM UTC

You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...

Re: Using AWK and R to parse 25TB

#20

Post by malshe » Thu, Jun 27, 2019, 2:29 PM UTC

stirfrykitty wrote: post #11

There was a similar article (2014) that is also interesting. I think too many of us see new and shiny and immediately glom onto it, forgetting that the UNIX/regex fathers knew a thing or two about crunching data. https://adamdrake.com/command-line-tools-can-be-235x-faster-...

Thanks for the link. Very interesting.

In OP's article there is a link to a book "Data Science at the Command Line" which sounds quite relevant: https://www.datascienceatthecommandline.com