Live data from Hacker News

Big data in genomics: The $1k genome has arrived

nature.com

41–50 of 58 posts

Re: Big data in genomics: The $1k genome has arrived

#41

I think the currenntly more interesting application in sequencing is pathogen detection. These have smaller and simpler genomes (mostly), and tracking them and their features improves epidemiology and the choice of treatments.

Pathogen identification is indeed a very exciting application for NGS. In case you're interested, here[0] is a paper about a tool called SURPI (Sequence-based Ultra-Rapid Pathogen Identification) which was designed for that purpose. Also, here[1] is a case report from the NEJM where SURPI was used to diagnose a patient with Neuroleptospirosis, which allowed him to be treated quickly and eventually recover. SURPI isn'…

The clinical work they're doing is great, but the code is problematic. Its a bunch of Perl and Python duct taped together with shell scripts. From the github repo:

Shell 84.3% Perl 8.9% Python 6.3% C 0.5%

Check out the source https://github.com/chiulab/surpi

Re: Big data in genomics: The $1k genome has arrived

#42
post #6

The bottleneck in genomics hasn't been cost since about 2012. The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well. I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on l…

You know... I'm not so sure I completely agree. I mean, yes, the analysis is the bottleneck. Since about 2012 it takes more time to perform the analysis than it does to generate the data. And even though all of the raw data -> BAM -> VCF or counts steps is automated, nothing after that is (or can be, since that's where the science happens).

But, the $1k genome really is significant. Costs matter.

Researchers will only sequence what they can afford. So, when sequencing a genome was $10k, there weren't too many 30X whole genomes sequenced. Instead we had exome sequencing of individuals or low-depth sequencing of populations.

Fast forward a few years, and now the costs have gone down significantly. Because of this, we are now starting to see more whole-genomes. Now, the costs of to sequence a whole-genome at 30X is about the same as targeted exome sequencing (due to the extra capture costs). If it's almost cheaper to just sequence the whole-genome and skip any extra capture probe biases, why not just sequence the whole thing? This is particularly true when looking at complex diseases where the underlying cause(s) may not be in the coding part of a gene.

So, in that regard, cost really was a bottleneck. With all of the new whole-genomes available, we can start to ask even tougher questions. And with more data available from more people, we can start to get some real confidence behind our predictions.

But, you are right in that it does all lead back to the data-analysis bottleneck. Before we had as more data than we could deal with. Now, we can process data faster, so that's good. But we also have significantly more data to process, so in the end, it's about even.

The other issue then becomes data storage. What do you do when you have a few hundred terabytes or a petabyte of sequencing data? S3 can get pretty expensive at that scale. Instead, it's probably just cheaper to store the DNA in a freezer, and if you ever need the data - just resequence it.

Edit: reworded a bit

Re: Big data in genomics: The $1k genome has arrived

#43
post #8

A quick search for public genome data led me here http://www.completegenomics.com/public-data/69-Genomes/ . I wonder if there will be a time when you can search millions through a rich UI, comparing your own genome with others, etc.

For more genomic data check out www.solvebio.com

You'll need to register for a (free) API key to access public data, although we're removing that requirement in the next few days (I work here.)

Re: Big data in genomics: The $1k genome has arrived

#44
post #42
post #6

The bottleneck in genomics hasn't been cost since about 2012. The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well. I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on l…

You know... I'm not so sure I completely agree. I mean, yes, the analysis is the bottleneck. Since about 2012 it takes more time to perform the analysis than it does to generate the data. And even though all of the raw data -> BAM -> VCF or counts steps is automated, nothing after that is (or can be, since that's where the science happens). But, the $1k genome really is significant. Costs matter. Researchers will onl…

Compute time and storage costs are also bottlenecks, so even though the base calling through VCF is automated large genome centers struggle to keep up with the output of the new sequencers

Re: Big data in genomics: The $1k genome has arrived

#45

Earlier quoted context omitted.

How long does it take (and with what computational bandwidth) to produce a 125MB variant file from 200GB raw sequence data?

Depending on the pipeline you use and the compute resources available you could have a full workflow done in anywhere from several hours to a couple days. Illumina BaseSpace is free (for now) and has some example data sets with a bunch of canned pipelines for analysis if you're interested in trying it for yourself. https://basespace.illumina.com/

You're not going to VCF on a whole genome in several hours.

Re: Big data in genomics: The $1k genome has arrived

#46
post #45

Earlier quoted context omitted.

Depending on the pipeline you use and the compute resources available you could have a full workflow done in anywhere from several hours to a couple days. Illumina BaseSpace is free (for now) and has some example data sets with a bunch of canned pipelines for analysis if you're interested in trying it for yourself. https://basespace.illumina.com/

You're not going to VCF on a whole genome in several hours.

Could you please elaborate on this?

Re: Big data in genomics: The $1k genome has arrived

#47
post #44
post #42

Earlier quoted context omitted.

You know... I'm not so sure I completely agree. I mean, yes, the analysis is the bottleneck. Since about 2012 it takes more time to perform the analysis than it does to generate the data. And even though all of the raw data -> BAM -> VCF or counts steps is automated, nothing after that is (or can be, since that's where the science happens). But, the $1k genome really is significant. Costs matter. Researchers will onl…

Compute time and storage costs are also bottlenecks, so even though the base calling through VCF is automated large genome centers struggle to keep up with the output of the new sequencers

When you say compute time, do you have in mind purely quantitative processing/analytics, or streaming and data transfer?

Is there any way you could describe a sample "experiment" and the associated storage costs and compute time, just a paper-and-pencil estimation perhaps?

Re: Big data in genomics: The $1k genome has arrived

#48

Earlier quoted context omitted.

Is the time required to upload data to the cloud ever a problem with these solutions? Of course, it depends on what you are trying to do, but suppose you were working with thousands of genomes?

Sure, all the time. Network and I/O are the biggest blockers for sequence analysis. For any organization that is working with thousands of genomes they probably have their own compute resources. I know of at least one organization who is currently sending thousands of genomes to the cloud for analysis, so it's certainly feasible to some extent.

Any chance we could follow up on this? I'm conducting research in this space and would love to have a chat regarding barriers to such large-scale analysis.

If you could message me at hngenometemp@forward.cat that would be terrific!

Re: Big data in genomics: The $1k genome has arrived

#49
post #45

Earlier quoted context omitted.

Depending on the pipeline you use and the compute resources available you could have a full workflow done in anywhere from several hours to a couple days. Illumina BaseSpace is free (for now) and has some example data sets with a bunch of canned pipelines for analysis if you're interested in trying it for yourself. https://basespace.illumina.com/

You're not going to VCF on a whole genome in several hours.

With particular hardware and software you can. Edico Dragen claims speeds for bcl -> vcf of 20 minutes [1]. With Microsoft Research's snap aligner and 450GB of memory you can get whole genome alignment in ~30 minutes and then variant calling can be done in a couple hours.

1. http://www.edicogenome.com/dragen/dragen-gp/

Re: Big data in genomics: The $1k genome has arrived

#50
post #6

The bottleneck in genomics hasn't been cost since about 2012. The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well. I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on l…

People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open source software) that you can pay for. DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. Unless you are doing a custom prep for your sequencing one of these probably has an analysis solution for you.

All of these will give you some answers. The reality is there is SO MUCH MORE INFORMATION to be gleaned from our genomes that these solutions do not account for.
Post reply on HN