Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using AWK and R to parse 25TB

livefreeordichotomize.com

Post reply on HN
Print view View original

81–90 of 106 posts

Re: Using AWK and R to parse 25TB

#81

Post by devonkim »

dgacmu wrote: post #77
devonkim wrote: post #75

Earlier quoted context omitted.

I watched a talk by someone in the intelligence community space nearly 8 years ago talking about the data dirt that most companies and spy agencies are combing through and the kind of abstract research that will be necessary to turn that into something consumable by all the stuff that private sector seems to be selling and hyping. So I think the old guard big data folks collecting yottabytes of crap across the world…

> In academia, who the heck gets a doctorate for advancements in cleaning up arbitrary data to feed into ML models Well - Alex Ratner [stanford], for one: https://ajratner.github.io/ And several of Chris Re's other students have as well: https://cs.stanford.edu/~chrismre/ Trifacta is Joseph Hellerstein's [berkeley] startup for data wrangling: https://www.trifacta.com/ Sanjay Krishnan [berkeley]: http://sanjayk.io/

I was asking somewhat rhetorically but am glad to see that there’s some serious efforts going into weak supervision. At the risk of goalpost moving, I am curious who besides those in the Bay Area at the cutting edge are working on this pervasive problem? My more substantive point is that given the massive data quality problem among the ML community I would expect these researchers to be superhero class but why aren’t they?

Re: Using AWK and R to parse 25TB

#82

Post by IndustrialJane »

emmanueloga_ wrote: post #78

Unix pipelines, AWK, gnu parallel, R, all great stuff. If you have such an specific task, why not just write an "actual program" (as opposed to pipeline of scripts)? From the looks of it, it sounds like this problem could have been solved with, say, 50 lines of Java, C, Go, etc, etc. Maybe a bit more verbose but it would give you full control, you wouldn't need to lookup how to use command line parameters on S/O, and…

I have done that more than once. I often end up with a solution that works on the test set but which breaks after 10 TB just because is a valid email address according RFC-822 (Who the f* thought it was a good idea to allow spaces in email addresses?). Or some other exception that was not part of the test set, and that was not identified before starting.

Dealing with exceptions is extremely error prone if these exceptions are not mapped beforehand. Thus it can be very costly.

Similarly doing stuff in parallel is extremely error prone due to race conditions: What does not happen when running on your 1 GB test set, may very well happen when running on your 25 TB production data.

Re: Using AWK and R to parse 25TB

#83

Post by IndustrialJane »

nimrody wrote: post #74

Very well written. I think using 'make' with the -j parameter (# of parallel jobs) is more useful than using gnu 'parallel'. The reason is that if one of the job fails for some reason, you just re-run 'make' and only the required jobs are started instead of restarting the entire computation.

With --results or --joblog and --resume-failed GNU Parallel can do this, too.

Re: Using AWK and R to parse 25TB

#84

Post by IndustrialJane »

bsg75 wrote: post #26

Have used Mawk [1] in similar cases for a runtime savings, provided that the script works without any GNU Awk extensions. [1] https://invisible-island.net/mawk/

I always wondered why Mawk did not implement Gawk's extensions - or at least those that could be done without any major penalty for the rest of the code.

Do you know why?

Re: Using AWK and R to parse 25TB

#85

Post by dredmorbius »

sansnomme wrote: post #15

What's the best pipeline/workflow management tool for command line programs with a GUI? E.g. for resuming the process after it gets interrupted etc.

Screen, if it's the connection session itself. Tmux if you're new-school.

Task-splitting itself is inherently recoverable, as incomplete work units don't produce the final output you're looking for, and can. be retried, re-run, or re-entered into the task pool.

A GUI quite frankly simply gets in the way.

Re: Using AWK and R to parse 25TB

#87

Post by [deleted] »

nimrody wrote: post #74

Very well written. I think using 'make' with the -j parameter (# of parallel jobs) is more useful than using gnu 'parallel'. The reason is that if one of the job fails for some reason, you just re-run 'make' and only the required jobs are started instead of restarting the entire computation.

[deleted]

Re: Using AWK and R to parse 25TB

#88

Post by emmanueloga_ »

IndustrialJane wrote: post #82
emmanueloga_ wrote: post #78

Unix pipelines, AWK, gnu parallel, R, all great stuff. If you have such an specific task, why not just write an "actual program" (as opposed to pipeline of scripts)? From the looks of it, it sounds like this problem could have been solved with, say, 50 lines of Java, C, Go, etc, etc. Maybe a bit more verbose but it would give you full control, you wouldn't need to lookup how to use command line parameters on S/O, and…

I have done that more than once. I often end up with a solution that works on the test set but which breaks after 10 TB just because is a valid email address according RFC-822 (Who the f * thought it was a good idea to allow spaces in email addresses?). Or some other exception that was not part of the test set, and that was not identified before starting. Dealing with exceptions is extremely error prone if these exce…

I get your point but the same error handling problems can appear in scripts and pipelines, no?

In a program I'd try/catch defensively "just in case", if missing one line out of 25TB is not a bit deal.

For parallel processing I'd reach for the nearest standard library at hand on the language of choice.

Re: Using AWK and R to parse 25TB

#89

Post by danielecook »

Here is an alternative solution, although yours is a good one.

If you only need a single SNP or a group of SNPs within a region, you can use tabix[1] to index gzipped TSVs and query by genomic position. The position of SNPs can be obtained from a lookup table (2.5M is not very big even for R) or from an API if you were to say - query by rsid.

tabix also works over http (and s3) and can utilize RANGE queries to select a subset of a file...so you only wind up downloading a or reading a small portion once it is indexed and can do something like this:

tabix https://www.file.url.tsv chr1:1-1000

The command above would return variants on chromosome 1 between 1 and 1000.

The following variant browser works in this way: https://elegansvariation.org/data/browser/ - Theres no formal database (e.g. MySQL) running here, just tabix (actually bcftools which uses tabix) to select variants in a particular region, wrap them in JSON, and return to the client.

Setting this up on S3 requires configuring CORS... the igv browser also uses tabix indexes and provides guidance on how to set this up [2]

[1] https://www.htslib.org/doc/tabix.html [2] https://github.com/igvteam/igv.js/wiki/Data-Server-Requireme...

I created a similar solution to what you have done using this alternative approach, writing a wrapper in R that invoked bcftools under the hood. The dataset I was working with was a lot smaller (1.6M Snps x 252 individuals), but should work with larger genotype sets as well.

Post reply on HN