Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data in genomics: The $1k genome has arrived

nature.com

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Big data in genomics: The $1k genome has arrived

#21

Post by epistasis »

BKPetkov wrote: post #17
adenadel wrote: post #10

Earlier quoted context omitted.

People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open source software) that you can pay for. DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. Unless you are doing a custom prep for your sequencing one of these probably has an analysis solution for you.

Is the time required to upload data to the cloud ever a problem with these solutions? Of course, it depends on what you are trying to do, but suppose you were working with thousands of genomes?

The sequencers can stream to a data analysis center as the data is being generated.

It takes a 100mbit stream/$1M of sequencing capital, so network connectivity to transfer to a data center is a tiny tiny cost of the whole ordeal.

However, paying for AWS storage is pretty prohibitive, unless you're at a small scale. So big centers will build their own storage facilities.

The small data producers like the ones that the thread author talks about can use often use AWS more cost efficiently than building a compute cluster. However, they need to budget for that, which is not always thought of. They may also need to fight their institute's core center so that they can use DNANexus.

Re: Big data in genomics: The $1k genome has arrived

#22

Post by [deleted] »

BKPetkov wrote: post #12
A_Beer_Clinked wrote: post #3

Earlier quoted context omitted.

I found this link: https://medium.com/precision-medicine/how-big-is-the-human-g... In summary: > 1. In a perfect world (just your 3 billion letters): ~700 megabytes > 2. In the real world, right off the genome sequencer: ~200 gigabytes > 3. As a variant file, with just the list of mutations: ~125 megabytes

How long does it take (and with what computational bandwidth) to produce a 125MB variant file from 200GB raw sequence data?

[deleted]

Re: Big data in genomics: The $1k genome has arrived

#23

Post by [deleted] »

arca_vorago wrote: post #20
BKPetkov wrote: post #12

Earlier quoted context omitted.

How long does it take (and with what computational bandwidth) to produce a 125MB variant file from 200GB raw sequence data?

I've seen 200GB runs take 4 days, I've seen runs take 3 hours. Depends on your computing structure but more importantly is your IO. High CPU core counts and high speed storage access make a big difference, as does distributing the computational workload.

[deleted]

Re: Big data in genomics: The $1k genome has arrived

#24

Post by jerven »

adenadel wrote: post #10
searine wrote: post #6

The bottleneck in genomics hasn't been cost since about 2012. The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well. I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on l…

People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open source software) that you can pay for. DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. Unless you are doing a custom prep for your sequencing one of these probably has an analysis solution for you.

Analysis is custom if you are doing genomics right at this time. Either you are doing something pre-clinical, and you better understand what you are doing i.e. analysis in a black box is not ok. Or you are not dealing with human stuff at all and then there is no analysis in a box on the market.

There are tools with GUI's but they don't help that much if you don't understand the underlying systems.

Re: Big data in genomics: The $1k genome has arrived

#25

Post by jerven »

adenadel wrote: post #16
noname123 wrote: post #14

Earlier quoted context omitted.

>Though that $1000 produces 100GB of data Not sure if you are referring to the raw sequence data coming from the sequencers, nowadays the standard practice is to align a specific sample's genome (e.g., an individual's genome) to the reference genome (the Human Genome Project) and store only your variations against that reference genome, stored as a BAM (binary alignment) file. Furthermore, for most clinical cases, if…

You've got a bit of misinformation in here. He's correct about the raw sequence data for a 30X whole human genome being 100GB of data. The standard practice is to align the sample to the reference, but we store alignments in the BAM file, not variations. Some of the mismatches in the alignments are due to noise in the sequencer. We run the BAM through a variant caller which outputs a VCF which contains the variants.…

Of course its a VCF file which we hope contains the variants ;( the tech is still fickle and so is some of the biology...

Soon we are going to go towards variant graphs and VCFs will disappear again, slowly (IMHO).

Re: Big data in genomics: The $1k genome has arrived

#26

Post by jerven »

bayesianhorse wrote: post #7
epistasis wrote: post #2

Though that $1000 produces 100GB of data, after processing there's probably only about ~100MB of features left for machine learning, at most. And most of that will be incidental. With enough data we can hope to have a better filter between signal and noise. Until now, biology has had a huge problem that most big data settings don't: far far more features than labels. With enough patients' data, the matrix will become…

It's hard to quantify how much data is left after "analysis". An assembled genome sequence, at least a certain quality, is actually useful beyond machine learning. For example loss-of-function mutations can be detected without resorting to other genomes beyond the reference. Inversions, translocations and copy number variations can also give clues to illness.

If for 50% of the genes (human) we have no idea of function, then its hard to determine at this point what is a loss-of-function mutation in the first place.

Source: I am the lead web developer for uniprot.org, and I deal with lack of real info daily as does everyone in the Swiss-Prot team.

Re: Big data in genomics: The $1k genome has arrived

#27

Post by searine »

adenadel wrote: post #10
searine wrote: post #6

The bottleneck in genomics hasn't been cost since about 2012. The chokepoint is analysis. Most biologists don't know how to program, and most programmers don't know biological context, neither know statistics well. I work with so many scientists whose only thought is to sequence first and ask questions later. Usually all the real work ends up falling on the shoulders of one skilled researcher while the rest look on l…

People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open source software) that you can pay for. DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. Unless you are doing a custom prep for your sequencing one of these probably has an analysis solution for you.

>DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis.

"Analysis".

There is a difference between meaningful analysis which answers long standing questions of biology, and running a program which calls SNPs and spits out genome stats.

The hard part is knowing what questions to ask, and how to interpret the answers when you get them. That takes years of training to develop a critical mind.

Re: Big data in genomics: The $1k genome has arrived

#28

Post by gbveiuwlbiu »

jerven wrote: post #26
bayesianhorse wrote: post #7

Earlier quoted context omitted.

It's hard to quantify how much data is left after "analysis". An assembled genome sequence, at least a certain quality, is actually useful beyond machine learning. For example loss-of-function mutations can be detected without resorting to other genomes beyond the reference. Inversions, translocations and copy number variations can also give clues to illness.

If for 50% of the genes (human) we have no idea of function, then its hard to determine at this point what is a loss-of-function mutation in the first place. Source: I am the lead web developer for uniprot.org, and I deal with lack of real info daily as does everyone in the Swiss-Prot team.

Apart from data privacy and formal restrictions on sharing, what would you say the reason for this lack of public data is?

Am I right to say that the bandwidth and storage needed to upload whole genomes is prohibitive?

Re: Big data in genomics: The $1k genome has arrived

#29

Post by gbveiuwlbiu »

jerven wrote: post #26
bayesianhorse wrote: post #7

Earlier quoted context omitted.

It's hard to quantify how much data is left after "analysis". An assembled genome sequence, at least a certain quality, is actually useful beyond machine learning. For example loss-of-function mutations can be detected without resorting to other genomes beyond the reference. Inversions, translocations and copy number variations can also give clues to illness.

If for 50% of the genes (human) we have no idea of function, then its hard to determine at this point what is a loss-of-function mutation in the first place. Source: I am the lead web developer for uniprot.org, and I deal with lack of real info daily as does everyone in the Swiss-Prot team.

Actually, can I contact you @uniprot.org with more questions?

Re: Big data in genomics: The $1k genome has arrived

#30

Post by damurdock »

bayesianhorse wrote: post #9

I think the currenntly more interesting application in sequencing is pathogen detection. These have smaller and simpler genomes (mostly), and tracking them and their features improves epidemiology and the choice of treatments.

Pathogen identification is indeed a very exciting application for NGS. In case you're interested, here[0] is a paper about a tool called SURPI (Sequence-based Ultra-Rapid Pathogen Identification) which was designed for that purpose. Also, here[1] is a case report from the NEJM where SURPI was used to diagnose a patient with Neuroleptospirosis, which allowed him to be treated quickly and eventually recover. SURPI isn't the only horse in this game, of course, but I've worked with it before so it immediately came to mind.

[0]: "A cloud-compatible bioinformatics pipeline for ultrarapid pathogen identification from next-generation sequencing of clinical samples" http://genome.cshlp.org/content/24/7/1180.long

[1]: "Actionable Diagnosis of Neuroleptospirosis by Next-Generation Sequencing" http://www.nejm.org/doi/full/10.1056/NEJMoa1401268

Post reply on HN