Using AWK and R to parse 25TB
livefreeordichotomize.com
Using AWK and R to parse 25TB
1–10 of 106 posts
Re: Using AWK and R to parse 25TB
#2One note, this type of data isn't traditionally considered "sequencing" data, as it's a much simpler form of point measurements. Yes, you get the base call at a place of common variation, but "sequencing" is generally reserved for those cases where one gets extended sequences of bases, rather that just data from here and there.
Re: Using AWK and R to parse 25TB
#3I found this article particularly interesting as the author discusses a lot of (failed) methods.
Since I never dealt with big data before, I'm wondering what would you do in this situation?
Re: Using AWK and R to parse 25TB
#4Great post and thanks for sharing your learnings.
A couple of quick questions:
Was the 25TB raw data gathered from a single human genome?
What would be the size in bytes of a unique genomic fingerprint once raw data is all fully processed into high confidence base values? (including non-coding regions)
If we just look at coding regions and further compress by only looking at SNPs, how many bytes is that?
Considering that each base has ~2B of information... it would be super interesting to know how much space it takes to describe our uniqueness!
Re: Using AWK and R to parse 25TB
#5Great post and thanks for sharing your learnings. A couple of quick questions: Was the 25TB raw data gathered from a single human genome? What would be the size in bytes of a unique genomic fingerprint once raw data is all fully processed into high confidence base values? (including non-coding regions) If we just look at coding regions and further compress by only looking at SNPs, how many bytes is that? Considering…
Sorry I saw this article and thought it was pretty interesting. This is NOT my article but I'd like to know what others would do under this situation.
BTW not sure, but is it OK to post other's article here? Maybe I should add a short commentary in the title.
Re: Using AWK and R to parse 25TB
#6Great post and thanks for sharing your learnings. A couple of quick questions: Was the 25TB raw data gathered from a single human genome? What would be the size in bytes of a unique genomic fingerprint once raw data is all fully processed into high confidence base values? (including non-coding regions) If we just look at coding regions and further compress by only looking at SNPs, how many bytes is that? Considering…
Sorry I saw this article and thought it was pretty interesting. This is NOT my article but I'd like to know what others would do under this situation. BTW not sure, but is it OK to post other's article here? Maybe I should add a short commentary in the title.
I think it's fine. The author might show up otherwise others with interest in the area might reply.
Re: Using AWK and R to parse 25TB
#7Great post and thanks for sharing your learnings. A couple of quick questions: Was the 25TB raw data gathered from a single human genome? What would be the size in bytes of a unique genomic fingerprint once raw data is all fully processed into high confidence base values? (including non-coding regions) If we just look at coding regions and further compress by only looking at SNPs, how many bytes is that? Considering…
Q: "Was the 25TB raw data gathered from a single human genome?"
Essay: "Each row contained a data for a single SNP for a single person." ... "There were ~2.5 million SNPS and ~60 thousand people"
Statement: "it would be super interesting to know how much space it takes to describe our uniqueness"
Genome size is "3,234.83 Mb (Mega-basepairs) per haploid genome" says https://en.wikipedia.org/wiki/Human_genome .
However, question of uniqueness depends on your model. We are all unique.
If you have human genomes, then use a reference template and only list those which differ. Eg, https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc... .
Re: Using AWK and R to parse 25TB
#8Great post and thanks for sharing your learnings. A couple of quick questions: Was the 25TB raw data gathered from a single human genome? What would be the size in bytes of a unique genomic fingerprint once raw data is all fully processed into high confidence base values? (including non-coding regions) If we just look at coding regions and further compress by only looking at SNPs, how many bytes is that? Considering…
Sorry I saw this article and thought it was pretty interesting. This is NOT my article but I'd like to know what others would do under this situation. BTW not sure, but is it OK to post other's article here? Maybe I should add a short commentary in the title.
You don't even need to say it's not yours, probably most that is posted here isn't from the author.
Re: Using AWK and R to parse 25TB
#9Welcome to Bioinformatics! Excellent write up!
Re: Using AWK and R to parse 25TB
#10I found this article particularly interesting as the author discusses a lot of (failed) methods. Since I never dealt with big data before, I'm wondering what would you do in this situation?
It seems he had a lot of issues due to Spark executors failing which seems a setting issue. My guess is that the executors were being killed by the system OOM killer. Spark's memory management is counter-intuitive. Spark spills intelligently to disk so executors don't need a lot of memory to process data if you're not doing interactive queries. However spark will use all the memory it's given and sometimes it will use more than that (might be the OS actually, not sure). So the trick is to give Spark's executors LESS memory (as a percentage of the node's memory) so there's a buffer in case Spark uses more memory than allocated.