Big data in genomics: The $1k genome has arrived • Hacker News Board

Re: Big data in genomics: The $1k genome has arrived

#51

Post by laurencerowe » Thu, Nov 05, 2015, 9:57 PM UTC

epistasis wrote: post #21

BKPetkov wrote: post #17

Earlier quoted context omitted.

Is the time required to upload data to the cloud ever a problem with these solutions? Of course, it depends on what you are trying to do, but suppose you were working with thousands of genomes?

The sequencers can stream to a data analysis center as the data is being generated. It takes a 100mbit stream/$1M of sequencing capital, so network connectivity to transfer to a data center is a tiny tiny cost of the whole ordeal. However, paying for AWS storage is pretty prohibitive, unless you're at a small scale. So big centers will build their own storage facilities. The small data producers like the ones that th…

S3 storage is pretty cheap, it's the data egress that really costs.

For academic centers though there is often an incentive to move things in house due to different treatment for capital expenditures and the opportunity to externalize some of your costs from your grant onto central services.

Re: Big data in genomics: The $1k genome has arrived

#52

Post by zitterbewegung » Thu, Nov 05, 2015, 10:29 PM UTC

gbveiuwlbiu wrote: post #47

jghn wrote: post #44

Earlier quoted context omitted.

Compute time and storage costs are also bottlenecks, so even though the base calling through VCF is automated large genome centers struggle to keep up with the output of the new sequencers

When you say compute time, do you have in mind purely quantitative processing/analytics, or streaming and data transfer? Is there any way you could describe a sample "experiment" and the associated storage costs and compute time, just a paper-and-pencil estimation perhaps?

I went to a python talk where the guy did genomics . I asked him how long his program would take and he said I'll tell you when it's done. This means his analysis programs can take days to weeks and he really didn't know how long the current one can take .

Re: Big data in genomics: The $1k genome has arrived

#53

Post by epistasis » Fri, Nov 06, 2015, 1:29 AM UTC

laurencerowe wrote: post #51

epistasis wrote: post #21

Earlier quoted context omitted.

The sequencers can stream to a data analysis center as the data is being generated. It takes a 100mbit stream/$1M of sequencing capital, so network connectivity to transfer to a data center is a tiny tiny cost of the whole ordeal. However, paying for AWS storage is pretty prohibitive, unless you're at a small scale. So big centers will build their own storage facilities. The small data producers like the ones that th…

S3 storage is pretty cheap, it's the data egress that really costs. For academic centers though there is often an incentive to move things in house due to different treatment for capital expenditures and the opportunity to externalize some of your costs from your grant onto central services.

Data transfer is less than a single year of Glacier storage, so while it's pricy I wouldn't egress a major portion of the cost.

Keeping this data for less than 5-10 years is pretty questionable, since it's so expensive to generate. Eventually it may be cheaper to store the DNA and resequence when if it needs to be looked at again. However, if you're doing petabytes of storage, it's going to me much more economical to have your own storage and compute than to use AWS. Particularly at the rate that academic centers pay for sysadmins.

Re: Big data in genomics: The $1k genome has arrived

#54

Post by laurencerowe » Fri, Nov 06, 2015, 3:01 AM UTC

epistasis wrote: post #53

laurencerowe wrote: post #51

Earlier quoted context omitted.

S3 storage is pretty cheap, it's the data egress that really costs. For academic centers though there is often an incentive to move things in house due to different treatment for capital expenditures and the opportunity to externalize some of your costs from your grant onto central services.

Data transfer is less than a single year of Glacier storage, so while it's pricy I wouldn't egress a major portion of the cost. Keeping this data for less than 5-10 years is pretty questionable, since it's so expensive to generate. Eventually it may be cheaper to store the DNA and resequence when if it needs to be looked at again. However, if you're doing petabytes of storage, it's going to me much more economical to…

Running a public data portal our egress is higher than our storage costs. (We now proxy downloads through a direct connect to our university network...)

Remember to account for future reductions in storage costs. S3 has come down from $0.1500/GB month in 2010 to $0.0300/GB month today. And the recently introduced infrequent access storage tier is under half that again at $0.0125/GB month. It's now significantly cheaper to use S3/Azure/Google than running the storage ourselves.

Re: Big data in genomics: The $1k genome has arrived

#55

Post by bayesianhorse » Fri, Nov 06, 2015, 4:59 AM UTC

jerven wrote: post #26

bayesianhorse wrote: post #7

Earlier quoted context omitted.

It's hard to quantify how much data is left after "analysis". An assembled genome sequence, at least a certain quality, is actually useful beyond machine learning. For example loss-of-function mutations can be detected without resorting to other genomes beyond the reference. Inversions, translocations and copy number variations can also give clues to illness.

If for 50% of the genes (human) we have no idea of function, then its hard to determine at this point what is a loss-of-function mutation in the first place. Source: I am the lead web developer for uniprot.org, and I deal with lack of real info daily as does everyone in the Swiss-Prot team.

Actually, when a patient with a mysterious illness lacks a functional version for a particular protein which healthy members of his family do have, this is a valuable clue. It narrows down the search significantly, and there have been case reports on this.

Of course, not every mysterious illness can be solved this way...

Re: Big data in genomics: The $1k genome has arrived

#56

Post by a_bonobo » Fri, Nov 06, 2015, 7:22 AM UTC

mbreese wrote: post #42

searine wrote: post #6

The bottleneck in genomics hasn't been cost since about 2012. The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well. I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on l…

You know... I'm not so sure I completely agree. I mean, yes, the analysis is the bottleneck. Since about 2012 it takes more time to perform the analysis than it does to generate the data. And even though all of the raw data -> BAM -> VCF or counts steps is automated, nothing after that is (or can be, since that's where the science happens). But, the $1k genome really is significant. Costs matter. Researchers will onl…

>With all of the new whole-genomes available, we can start to ask even tougher questions. And with more data available from more people, we can start to get some real confidence behind our predictions.

What I'd like to add to this is that we can spend more time and less money on "hobby" projects!

These are the projects you don't really have explicit funding for, but you do anyway, using leftover money from various pots [1]. We have two PhD students who work on "hobby" projects.

These projects include "outlandish" questions or "basic" research the funding committees didn't see the reason to fund.

In our case, these two PhD students work with less studied plant organisms no-one else really looked into so far, but which have some interesting attributes. Some of these hobbies have been fruitful (interesting novel salinity resistance proteins found), some of these less (but it's always good to populate public databases from data of less well-known organisms).

Cheaper genomics allow us to do so much more of these projects, exciting times!

[1] See http://www.phdcomics.com/comics.php?f=1431

(Of course, now the cost bottlenecks move somewhere else - it has become cheap to sequence an entire plant population, but it's still expensive to grow & especially phenotype that population - if you're into computer vision and are looking for a new project, automatic plant phenotyping is a field that still has a lot of work to do!)

Re: Big data in genomics: The $1k genome has arrived

#57

Post by akuchling » Fri, Nov 06, 2015, 3:08 PM UTC

zitterbewegung wrote: post #52

gbveiuwlbiu wrote: post #47

Earlier quoted context omitted.

When you say compute time, do you have in mind purely quantitative processing/analytics, or streaming and data transfer? Is there any way you could describe a sample "experiment" and the associated storage costs and compute time, just a paper-and-pencil estimation perhaps?

I went to a python talk where the guy did genomics . I asked him how long his program would take and he said I'll tell you when it's done. This means his analysis programs can take days to weeks and he really didn't know how long the current one can take .

Possibly C. Titus Brown? Here's his PyCon 2014 talk, "Data intensive biology in the cloud": https://www.youtube.com/watch?v=RCRbfKK57X8

Re: Big data in genomics: The $1k genome has arrived

#58

Post by mbreese » Fri, Nov 06, 2015, 3:11 PM UTC

jghn wrote: post #44

mbreese wrote: post #42

Earlier quoted context omitted.

You know... I'm not so sure I completely agree. I mean, yes, the analysis is the bottleneck. Since about 2012 it takes more time to perform the analysis than it does to generate the data. And even though all of the raw data -> BAM -> VCF or counts steps is automated, nothing after that is (or can be, since that's where the science happens). But, the $1k genome really is significant. Costs matter. Researchers will onl…

Compute time and storage costs are also bottlenecks, so even though the base calling through VCF is automated large genome centers struggle to keep up with the output of the new sequencers

Right - but we used to struggle with the output of the last generation of sequencers too. Computing capacity and storage have scaled at about the same rate. Or perhaps it's more accurate to say that the costs have come down, so we can throw more iron at the problem - these are embarrassingly parallel problems...

As an example, the last sequencing analysis cluster I setup (~2010) ran with 2x bonded gigabit networks between our compute nodes and storage and had 10TB of network storage. The newest one is running 10GigE, has 200+TB of storage, and is cheaper. And we are a small shop in this field.