Big data in genomics: The $1k genome has arrived
1–10 of 58 posts
Re: Big data in genomics: The $1k genome has arrived
#2Though that $1000 produces 100GB of data, after processing there's probably only about ~100MB of features left for machine learning, at most. And most of that will be incidental. With enough data we can hope to have a better filter between signal and noise.
Until now, biology has had a huge problem that most big data settings don't: far far more features than labels. With enough patients' data, the matrix will become squarish, but that's a long time from now, still.
Re: Big data in genomics: The $1k genome has arrived
#3Though that $1000 produces 100GB of data, after processing there's probably only about ~100MB of features left for machine learning, at most. And most of that will be incidental. With enough data we can hope to have a better filter between signal and noise. Until now, biology has had a huge problem that most big data settings don't: far far more features than labels. With enough patients' data, the matrix will become…
I found this link:
https://medium.com/precision-medicine/how-big-is-the-human-g...
In summary:
> 1. In a perfect world (just your 3 billion letters): ~700 megabytes
> 2. In the real world, right off the genome sequencer: ~200 gigabytes
> 3. As a variant file, with just the list of mutations: ~125 megabytes
Re: Big data in genomics: The $1k genome has arrived
#4Though that $1000 produces 100GB of data, after processing there's probably only about ~100MB of features left for machine learning, at most. And most of that will be incidental. With enough data we can hope to have a better filter between signal and noise. Until now, biology has had a huge problem that most big data settings don't: far far more features than labels. With enough patients' data, the matrix will become…
I found this link: https://medium.com/precision-medicine/how-big-is-the-human-g... In summary: > 1. In a perfect world (just your 3 billion letters): ~700 megabytes > 2. In the real world, right off the genome sequencer: ~200 gigabytes > 3. As a variant file, with just the list of mutations: ~125 megabytes
If there is only .1% variation then we should be able to get a diff down to ~1MB with some cleverness.
Re: Big data in genomics: The $1k genome has arrived
#5Earlier quoted context omitted.
I found this link: https://medium.com/precision-medicine/how-big-is-the-human-g... In summary: > 1. In a perfect world (just your 3 billion letters): ~700 megabytes > 2. In the real world, right off the genome sequencer: ~200 gigabytes > 3. As a variant file, with just the list of mutations: ~125 megabytes
If there is only .1% variation then we should be able to get a diff down to ~1MB with some cleverness.
That sounds quite feasible, though it hasn't really been worth the effort until we have quite a few more genomes. And typically extra information about the variant (is it in a gene, does it change a protein, etc.) so that extra lookups aren't required during a scan.
There are typically 4-6 million variants discovered through this method of genome sequencing in a normal genome. A simple variant consists of a genome coordinate at ~32 bits (one of 3.2e9), and the change from the reference, which is a x,y index into {A, C, G, T}^2, at ~4 bits. Typically the coordinates are spaced on average ~1k bases apart, so the coordinate could probably be squeezed into ~15bits with clever encoding. So a naive encoding of this information gets to 27MB, and that could probably be shoved down into 10MB if coordinates are deltas from the previous, rather than absolute. 1MB seems feasible, but with diminishing returns computationally.
Re: Big data in genomics: The $1k genome has arrived
#6The bottleneck in genomics hasn't been cost since about 2012.
The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well.
I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on like some unionized road crew.
It's only going to get worse, but the good news is, if you are one of the biologists who can program and use statistics then you're in good shape. There is already so much idle data out there already, that you'll never have to spend a dime on sequencing.
Re: Big data in genomics: The $1k genome has arrived
#7Though that $1000 produces 100GB of data, after processing there's probably only about ~100MB of features left for machine learning, at most. And most of that will be incidental. With enough data we can hope to have a better filter between signal and noise. Until now, biology has had a huge problem that most big data settings don't: far far more features than labels. With enough patients' data, the matrix will become…
It's hard to quantify how much data is left after "analysis". An assembled genome sequence, at least a certain quality, is actually useful beyond machine learning.
For example loss-of-function mutations can be detected without resorting to other genomes beyond the reference. Inversions, translocations and copy number variations can also give clues to illness.
Re: Big data in genomics: The $1k genome has arrived
#8A quick search for public genome data led me here http://www.completegenomics.com/public-data/69-Genomes/. I wonder if there will be a time when you can search millions through a rich UI, comparing your own genome with others, etc.
Re: Big data in genomics: The $1k genome has arrived
#9I think the currenntly more interesting application in sequencing is pathogen detection. These have smaller and simpler genomes (mostly), and tracking them and their features improves epidemiology and the choice of treatments.
Re: Big data in genomics: The $1k genome has arrived
#10The bottleneck in genomics hasn't been cost since about 2012. The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well. I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on l…
People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open source software) that you can pay for. DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. Unless you are doing a custom prep for your sequencing one of these probably has an analysis solution for you.