Big data in genomics: The $1k genome has arrived • Hacker News Board

Re: Big data in genomics: The $1k genome has arrived

#11

Post by noname123 » Thu, Nov 05, 2015, 5:41 PM UTC

bayesianhorse wrote: post #9

I think the currenntly more interesting application in sequencing is pathogen detection. These have smaller and simpler genomes (mostly), and tracking them and their features improves epidemiology and the choice of treatments.

nextflu.org and wwarn.org and ebola.nextfu.org

NextFlu takes global samples uploaded to a global flu genomic sample database and construct a phylogeny tree to monitor how influenza evolve and I think the authors behind the project wants to make predictions as to which cohort of influenza variation will become dominant.

Wwarn.org I believes tracks the emergence of Artesminin resistance (front-line drug in malaria treatment) in malaria in SE Asia and tries to map it out on GIS to inform public health officials further from SE Asia how it is spreading to their region (India, Africa where current Artesminin resistance gene is only 5% while Artesminin resistance is already the dominant wild type in SE Asia) and whether to modify front-line treatment protocol.

Re: Big data in genomics: The $1k genome has arrived

#12

Post by BKPetkov » Thu, Nov 05, 2015, 5:44 PM UTC

A_Beer_Clinked wrote: post #3

epistasis wrote: post #2

Though that $1000 produces 100GB of data, after processing there's probably only about ~100MB of features left for machine learning, at most. And most of that will be incidental. With enough data we can hope to have a better filter between signal and noise. Until now, biology has had a huge problem that most big data settings don't: far far more features than labels. With enough patients' data, the matrix will become…

I found this link: https://medium.com/precision-medicine/how-big-is-the-human-g... In summary: > 1. In a perfect world (just your 3 billion letters): ~700 megabytes > 2. In the real world, right off the genome sequencer: ~200 gigabytes > 3. As a variant file, with just the list of mutations: ~125 megabytes

How long does it take (and with what computational bandwidth) to produce a 125MB variant file from 200GB raw sequence data?

Re: Big data in genomics: The $1k genome has arrived

#13

Post by BKPetkov » Thu, Nov 05, 2015, 5:47 PM UTC

epistasis wrote: post #5

TheEzEzz wrote: post #4

Earlier quoted context omitted.

If there is only .1% variation then we should be able to get a diff down to ~1MB with some cleverness.

That sounds quite feasible, though it hasn't really been worth the effort until we have quite a few more genomes. And typically extra information about the variant (is it in a gene, does it change a protein, etc.) so that extra lookups aren't required during a scan. There are typically 4-6 million variants discovered through this method of genome sequencing in a normal genome. A simple variant consists of a genome co…

Interesting - wouldn't you need to have access to "the rest" of the genomes that you are comparing against? In other words, wouldn't you need to keep that ~100 GB from the newly sequenced genome in temporary storage while comparing against the rest of the database stored somewhere in the cloud, before then condensing the new genome into a variant file?

Re: Big data in genomics: The $1k genome has arrived

#14

Post by noname123 » Thu, Nov 05, 2015, 5:47 PM UTC

epistasis wrote: post #2

Though that $1000 produces 100GB of data, after processing there's probably only about ~100MB of features left for machine learning, at most. And most of that will be incidental. With enough data we can hope to have a better filter between signal and noise. Until now, biology has had a huge problem that most big data settings don't: far far more features than labels. With enough patients' data, the matrix will become…

>Though that $1000 produces 100GB of data

Not sure if you are referring to the raw sequence data coming from the sequencers, nowadays the standard practice is to align a specific sample's genome (e.g., an individual's genome) to the reference genome (the Human Genome Project) and store only your variations against that reference genome, stored as a BAM (binary alignment) file.

Furthermore, for most clinical cases, if you only care about the current known SNP (single nucleotide polymorphism in coding genes), you can generate a VCF file at those known SNP sites, compressing the data further. This is the way most population genetics is done by comparing the VCF files of cohorts, not doing analysis of single genomes of patients one by one.

Re: Big data in genomics: The $1k genome has arrived

#15

Post by arca_vorago » Thu, Nov 05, 2015, 5:52 PM UTC

searine wrote: post #6

The bottleneck in genomics hasn't been cost since about 2012. The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well. I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on l…

Having worked in a genetics lab as a sysadmin, I can agree with this 100%.

Re: Big data in genomics: The $1k genome has arrived

#16

Post by adenadel » Thu, Nov 05, 2015, 5:54 PM UTC

noname123 wrote: post #14

epistasis wrote: post #2

Though that $1000 produces 100GB of data, after processing there's probably only about ~100MB of features left for machine learning, at most. And most of that will be incidental. With enough data we can hope to have a better filter between signal and noise. Until now, biology has had a huge problem that most big data settings don't: far far more features than labels. With enough patients' data, the matrix will become…

>Though that $1000 produces 100GB of data Not sure if you are referring to the raw sequence data coming from the sequencers, nowadays the standard practice is to align a specific sample's genome (e.g., an individual's genome) to the reference genome (the Human Genome Project) and store only your variations against that reference genome, stored as a BAM (binary alignment) file. Furthermore, for most clinical cases, if…

You've got a bit of misinformation in here. He's correct about the raw sequence data for a 30X whole human genome being 100GB of data. The standard practice is to align the sample to the reference, but we store alignments in the BAM file, not variations. Some of the mismatches in the alignments are due to noise in the sequencer. We run the BAM through a variant caller which outputs a VCF which contains the variants. You do not typically genotype at only known SNP sites when generating a VCF file (although you may restrict by a region). If you were only interested in certain SNP sites you would be better off running a microarray.

Re: Big data in genomics: The $1k genome has arrived

#17

Post by BKPetkov » Thu, Nov 05, 2015, 5:59 PM UTC

adenadel wrote: post #10

searine wrote: post #6

The bottleneck in genomics hasn't been cost since about 2012. The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well. I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on l…

People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open source software) that you can pay for. DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. Unless you are doing a custom prep for your sequencing one of these probably has an analysis solution for you.

Is the time required to upload data to the cloud ever a problem with these solutions? Of course, it depends on what you are trying to do, but suppose you were working with thousands of genomes?

Re: Big data in genomics: The $1k genome has arrived

#18

Post by adenadel » Thu, Nov 05, 2015, 6:05 PM UTC

BKPetkov wrote: post #17

adenadel wrote: post #10

Earlier quoted context omitted.

People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open source software) that you can pay for. DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. Unless you are doing a custom prep for your sequencing one of these probably has an analysis solution for you.

Is the time required to upload data to the cloud ever a problem with these solutions? Of course, it depends on what you are trying to do, but suppose you were working with thousands of genomes?

Sure, all the time. Network and I/O are the biggest blockers for sequence analysis. For any organization that is working with thousands of genomes they probably have their own compute resources. I know of at least one organization who is currently sending thousands of genomes to the cloud for analysis, so it's certainly feasible to some extent.

Re: Big data in genomics: The $1k genome has arrived

#19

Post by adenadel » Thu, Nov 05, 2015, 6:08 PM UTC

BKPetkov wrote: post #12

A_Beer_Clinked wrote: post #3

Earlier quoted context omitted.

I found this link: https://medium.com/precision-medicine/how-big-is-the-human-g... In summary: > 1. In a perfect world (just your 3 billion letters): ~700 megabytes > 2. In the real world, right off the genome sequencer: ~200 gigabytes > 3. As a variant file, with just the list of mutations: ~125 megabytes

How long does it take (and with what computational bandwidth) to produce a 125MB variant file from 200GB raw sequence data?

Depending on the pipeline you use and the compute resources available you could have a full workflow done in anywhere from several hours to a couple days. Illumina BaseSpace is free (for now) and has some example data sets with a bunch of canned pipelines for analysis if you're interested in trying it for yourself. https://basespace.illumina.com/

Re: Big data in genomics: The $1k genome has arrived

#20

Post by arca_vorago » Thu, Nov 05, 2015, 6:09 PM UTC

BKPetkov wrote: post #12

A_Beer_Clinked wrote: post #3

Earlier quoted context omitted.

I found this link: https://medium.com/precision-medicine/how-big-is-the-human-g... In summary: > 1. In a perfect world (just your 3 billion letters): ~700 megabytes > 2. In the real world, right off the genome sequencer: ~200 gigabytes > 3. As a variant file, with just the list of mutations: ~125 megabytes

How long does it take (and with what computational bandwidth) to produce a 125MB variant file from 200GB raw sequence data?

I've seen 200GB runs take 4 days, I've seen runs take 3 hours. Depends on your computing structure but more importantly is your IO. High CPU core counts and high speed storage access make a big difference, as does distributing the computational workload.