Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using AWK and R to parse 25TB

livefreeordichotomize.com

Post reply on HN
Print view View original

51–60 of 106 posts

Re: Using AWK and R to parse 25TB

#51

Post by markus_zhang »

wongarsu wrote: post #24
scottlocklin wrote: post #19

You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...

For anything more complicated you can also get very far with simple python programs that read one line at a time and output some transformation of it (which might include turning one line into many to be piped into sort etc)

I think that the optimal way to do these kind of things is:

1) Assuming there is no joins/merges requirement, read in chunks and output GB dumps.

2) If joins/merges are required, use external merge sort.

Is this correct? Actually I'm wondering whether I could earn some bread and butter by focusing on the big data processing problems (e.g. sort/filter Terabytes+ dumps, do transformation for each line for Terabytes+ dumps, those kind of things) without actually knowing how to implement math algorithms (required for data science).

If so what kind of tools I need to master? I'm thinking about basic *nix tools like mentioned above, and also Python and maybe some compiled language for optimization (someone managed to speed up a Python external merge algorithm on 500GB file by 50% by implementing in Go), then maybe some easy algorithms (merge join, heap, etc.)

Re: Using AWK and R to parse 25TB

#52

Post by parhamn »

While I get the whole "you can do it on one node without all the complexity" thing, I do still wonder if map-reduce-synchronizer + coreutils is better than the behemoths that are the distributed ETL platforms right now. All the system would need to do is make a data file available on a node and capture stdout of the unix pipeline. I know gnu parallel does some of this.

Re: Using AWK and R to parse 25TB

#53

Post by markus_zhang »

malshe wrote: post #20
stirfrykitty wrote: post #11

There was a similar article (2014) that is also interesting. I think too many of us see new and shiny and immediately glom onto it, forgetting that the UNIX/regex fathers knew a thing or two about crunching data. https://adamdrake.com/command-line-tools-can-be-235x-faster-...

Thanks for the link. Very interesting. In OP's article there is a link to a book "Data Science at the Command Line" which sounds quite relevant: https://www.datascienceatthecommandline.com

Yeah I decide to read that book and install the Docker file. The thing is I'm kind of not sure how to setup the whole thing in Linux as I also need Python, some database and gcc. I think I should be able to find some tutorials for Ubuntu.

Re: Using AWK and R to parse 25TB

#54

Post by awiesenhofer »

devonkim wrote: post #48
scottlocklin wrote: post #19

You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...

Furthermore, as computers get faster and cheaper in every dimension what makes economic sense to use “Big Data” tooling and efforts gets substantially larger with it. The limits of single nodes 15 years ago were pretty serious but most problems businesses have even in the so-called enterprise can currently easily fit on a workstation costing maybe $5k and be crunched through in a couple hours or maybe minutes - a lot…

> most business leaders still seem to treat analytics results in discrete blocks via monthly / weekly reports and seem quite content with reports and findings that take hours to run.

I would go further and even call long or at least not instant report generation a perceived feature. Similar to flight and hotel booking sites that show some kind of loading screen even if they could give instant search results, the duration of the generation itself seems to add trust to the reports.

Re: Using AWK and R to parse 25TB

#55

Post by acomjean »

markus_zhang wrote: post #51
wongarsu wrote: post #24

Earlier quoted context omitted.

For anything more complicated you can also get very far with simple python programs that read one line at a time and output some transformation of it (which might include turning one line into many to be piped into sort etc)

I think that the optimal way to do these kind of things is: 1) Assuming there is no joins/merges requirement, read in chunks and output GB dumps. 2) If joins/merges are required, use external merge sort. Is this correct? Actually I'm wondering whether I could earn some bread and butter by focusing on the big data processing problems (e.g. sort/filter Terabytes+ dumps, do transformation for each line for Terabytes+ du…

Learning Unix tools is pretty good place to start. There are a lot of commands that can do a lot of processing. It’s been a while since I learned but the book “Unix power tools” from oreily is pretty good. It’s old, but honestly these commands haven’t changed much.

http://shop.oreilly.com/product/9780596003302.do

Python is slower compared to some of it’s compiled cousins, but it’s quick to write and a great skill to have when bash scripting can’t handle some of the complexity or you need dB access. We use it sometimes to call c programs to do DNA sequence alignments and process the returns.

Re: Using AWK and R to parse 25TB

#56

Post by jcastro »

nstrayer wrote: post #28

Hi! Author of the post here. I can attempt to answer any questions if need be although it looks like others have done a great job doing that already!

Love your writing style, I'm not anywhere close to working in this field and I learned something.

Re: Using AWK and R to parse 25TB

#57

Post by _hl_ »

usgroup wrote: post #40
_hl_ wrote: post #25

Earlier quoted context omitted.

> imagine GPU awk My intuition tells me that awk and other text processing tools won’t scale well to a GPGPU. I might be wrong though. Is there any example of something like grep etc working well on a GPU?

Pffff naysayers ... https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/c...

That's very surprising, assuming they didn't doctor the results by choosing the workload all too carefully.

I would have expected a GPU regex too perform much worse, given that regex matching is probably very branchy code. Especially since computation is generally way faster than IO.

Re: Using AWK and R to parse 25TB

#58

Post by markus_zhang »

nstrayer wrote: post #28

Hi! Author of the post here. I can attempt to answer any questions if need be although it looks like others have done a great job doing that already!

Hi nstrayer thanks for the excellent article!

I work as a data analyst and I never got to worry about big data as the DWH takes care of the aggregation for us, plus I only work in Windows.

I see now that it would be very useful to learn *nix tools in general, as it seems that the skills to process (not to predict/analyze) terabytes+ data are very valuable and expensive to acquire and could be one's butter and bread.

Re: Using AWK and R to parse 25TB

#59

Post by et2o »

wongarsu wrote: post #27
et2o wrote: post #16

My biggest question is why on earth did they give you 25 Tb of TSV genetic data? :-) I'm not sure what your sample was but seems like it would have been better to use one of the special binary file formats for genetic data. You wrote SNP chips, But in order to get to 25 Tb I assume there must be imputed calls, so it seems like a BGEN might have been a lot easier. This is speculation of course, I'm not sure exactly wh…

If you unpack all of https://files.pushshift.io/reddit/comments/ you have many Tb of JSONs that are just dumps of API responses that slowly change schema over the years. It's also an incredibly useful dataset. In the end CPUs are fast enough and compression algorithms good enough that I would argue it doesn't really matter what format you use for storage, as long as it's reasonably easy to read back.

In the case of genomics, there have been at this point decades of work developing high performance file formats and there are large ecosystems of tools around them. Lots of bioinformatics is really manipulating these files. So using a supported file format makes a big difference.

Re: Using AWK and R to parse 25TB

#60

Post by KingMachiavelli »

_hl_ wrote: post #25
usgroup wrote: post #22

Lol , this is basically how I roll most the time . However what Linux is really missing right now is command line tools that saturate a GPU. Just counterparts to all the favourites that utilise the GPU ... imagine GPU awk.

> imagine GPU awk My intuition tells me that awk and other text processing tools won’t scale well to a GPGPU. I might be wrong though. Is there any example of something like grep etc working well on a GPU?

When this concept was previously posted on HN, the top comment pointed out how it's the pipes that are inefficient when working on GPUs due to copying data from the CPU to the GPU and vice-versa for each command & pipe pair. I think even if we don't get pipes per se but I think we could expose GPGPU resources in a unix-like way but I suppose it depends on driver support.

But as to your question, a lot of traditional tools like grep, sed, aren't really suited for the GPU unless you are running them on a lot of files at once.

https://news.ycombinator.com/item?id=5803943

Post reply on HN