21–30 of 106 posts
You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...
Just counterparts to all the favourites that utilise the GPU ... imagine GPU awk.
Great post and thanks for sharing your learnings. A couple of quick questions: Was the 25TB raw data gathered from a single human genome? What would be the size in bytes of a unique genomic fingerprint once raw data is all fully processed into high confidence base values? (including non-coding regions) If we just look at coding regions and further compress by only looking at SNPs, how many bytes is that? Considering…
Describing your total unique genetic profile would obviously require a lot more space, and wouldn't be constant across individuals/ancestral backgrounds (e.g. there's more genetic diversity in people of African descent).
You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...
Lol , this is basically how I roll most the time . However what Linux is really missing right now is command line tools that saturate a GPU. Just counterparts to all the favourites that utilise the GPU ... imagine GPU awk.
My intuition tells me that awk and other text processing tools won’t scale well to a GPGPU. I might be wrong though. Is there any example of something like grep etc working well on a GPU?
My biggest question is why on earth did they give you 25 Tb of TSV genetic data? :-) I'm not sure what your sample was but seems like it would have been better to use one of the special binary file formats for genetic data. You wrote SNP chips, But in order to get to 25 Tb I assume there must be imputed calls, so it seems like a BGEN might have been a lot easier. This is speculation of course, I'm not sure exactly wh…
In the end CPUs are fast enough and compression algorithms good enough that I would argue it doesn't really matter what format you use for storage, as long as it's reasonably easy to read back.
My biggest question is why on earth did they give you 25 Tb of TSV genetic data? :-) I'm not sure what your sample was but seems like it would have been better to use one of the special binary file formats for genetic data. You wrote SNP chips, But in order to get to 25 Tb I assume there must be imputed calls, so it seems like a BGEN might have been a lot easier. This is speculation of course, I'm not sure exactly wh…
Great post and thanks for sharing your learnings. A couple of quick questions: Was the 25TB raw data gathered from a single human genome? What would be the size in bytes of a unique genomic fingerprint once raw data is all fully processed into high confidence base values? (including non-coding regions) If we just look at coding regions and further compress by only looking at SNPs, how many bytes is that? Considering…
Sorry I saw this article and thought it was pretty interesting. This is NOT my article but I'd like to know what others would do under this situation. BTW not sure, but is it OK to post other's article here? Maybe I should add a short commentary in the title.