Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Darwin: a genomics co-processor provides up to 15,000x acceleration

blog.acolyer.org

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#31

Post by tradedash »

CJefferson wrote: post #28
tradedash wrote: post #5

Serious question: if the processing problems are classified as NP, then couldn't that processing be outsourced to a blockchain? Why have miners solve for useless problems that have no long lasting impact like "number of zeros in a SHA string" instead of processing data such as this? I can easily see it being the case that the mining problem in a given blockchain could be based on real scientific problems that needs s…

One fundamental problem is that the problem you solve must in some way "sign" or validate the set of transactions you want to add to the Blockchain. You can't just solve anything and say "I did some work". So while we could hypothetically use NP complete problems (although there are several issues which arise to do with consistent difficulty), it's hard to do useful work, because we need the input to be fundamentally…

This answers my question. Thanks :D

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#32

Post by JohnJamesRambo »

h4l0 wrote: post #25
tradedash wrote: post #5

Serious question: if the processing problems are classified as NP, then couldn't that processing be outsourced to a blockchain? Why have miners solve for useless problems that have no long lasting impact like "number of zeros in a SHA string" instead of processing data such as this? I can easily see it being the case that the mining problem in a given blockchain could be based on real scientific problems that needs s…

Seeing this genuinely suprised me. I have been working on exactly the same topic that you have mentioned. Coinami[0], Coin-Application Mediator Interface is a blockchain application where scientific problems are distributed like grid and solvers are awarded by cryptocurrency. I will not go into detail but our first prototype was distributing read mapping to volunteers. Now, I will list the problems we have encountere…

Thank you for posting this. I learned a lot about Coinami that I didn’t know.

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#33

Post by JohnJamesRambo »

jacquesm wrote: post #17
tradedash wrote: post #5

Serious question: if the processing problems are classified as NP, then couldn't that processing be outsourced to a blockchain? Why have miners solve for useless problems that have no long lasting impact like "number of zeros in a SHA string" instead of processing data such as this? I can easily see it being the case that the mining problem in a given blockchain could be based on real scientific problems that needs s…

A blockchain is a data structure, it doesn't have processing power. Miners solve for nonsense problems because that's the simplest kind that you could solve in order to make the proposition work, anything on top of that requires more complexity. So it's not that it isn't possible, it's just that it would be charity from the point of view of the developers of the software to spend brain cycles on something they don't…

When Ethereum implements Proof of Stake it will be quite green. In my opinion this will be the death knell for wasteful PoW like bitcoin.

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#34

Post by tradedash »

h4l0 wrote: post #25
tradedash wrote: post #5

Serious question: if the processing problems are classified as NP, then couldn't that processing be outsourced to a blockchain? Why have miners solve for useless problems that have no long lasting impact like "number of zeros in a SHA string" instead of processing data such as this? I can easily see it being the case that the mining problem in a given blockchain could be based on real scientific problems that needs s…

Seeing this genuinely suprised me. I have been working on exactly the same topic that you have mentioned. Coinami[0], Coin-Application Mediator Interface is a blockchain application where scientific problems are distributed like grid and solvers are awarded by cryptocurrency. I will not go into detail but our first prototype was distributing read mapping to volunteers. Now, I will list the problems we have encountere…

Awesome! thanks for the reading material :D

1) The problem supply can be tackled in many many ways.

2) The verification problem is exactly why I stated NP as a problem class where the verification process is often relatively simple and straight forward. Obviously if that isn't the case then perhaps the problem isn't that well suited for this kind of environment.

3) The bandwidth issue is not something that I had considered. Makes absolute sense when you put it that way.

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#35

Post by ethelward »

tradedash wrote: post #34
h4l0 wrote: post #25

Earlier quoted context omitted.

Seeing this genuinely suprised me. I have been working on exactly the same topic that you have mentioned. Coinami[0], Coin-Application Mediator Interface is a blockchain application where scientific problems are distributed like grid and solvers are awarded by cryptocurrency. I will not go into detail but our first prototype was distributing read mapping to volunteers. Now, I will list the problems we have encountere…

Awesome! thanks for the reading material :D 1) The problem supply can be tackled in many many ways. 2) The verification problem is exactly why I stated NP as a problem class where the verification process is often relatively simple and straight forward. Obviously if that isn't the case then perhaps the problem isn't that well suited for this kind of environment. 3) The bandwidth issue is not something that I had cons…

> The verification problem is exactly why I stated NP as a problem class where the verification process is often relatively simple and straight forward

Just for reference, don't forget that NP behaviour is displayed asymptotically. Checking a solution can still be pretty costly.

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#36

Post by dekhn »

There have been a lot of co-processors in the genomics and sequencing world. They have all failed for obvious business reasons. It's not even clear that any genomics problem that exists today can't be solved with conventional hardware. Most genomics programs reach a few percent of the capacity of the hardware, so I think more software tuning and design is the right approach.

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#37

Post by searine »

dekhn wrote: post #36

There have been a lot of co-processors in the genomics and sequencing world. They have all failed for obvious business reasons. It's not even clear that any genomics problem that exists today can't be solved with conventional hardware. Most genomics programs reach a few percent of the capacity of the hardware, so I think more software tuning and design is the right approach.

I agree. The problem in genomics isn't how fast alignment is going, it is where to put the gobs and gobs of data.

The bottleneck in speed is how fast you can flip a DNA sequencer, which right now with an illumina Novaseq that bottleneck is a few days per run. That gives plenty of time to process each runs data on an HPC on current software for alignment and snp calling.

However, when each runfolder takes up 30Tb and you're flipping 20 runfolders a week. The scale of storage get large, fast.

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#38

Post by stochastic_monk »

klmr wrote: post #27

> The long read technology comes with a drawback though – high error rates in sequencing of between 15%-40% These error rate estimates are seriously outdated (the quoted number is from a 2015 paper but it was obsolete pretty much as soon as that paper was published). Long-read technologies are evolving rapidly, so this is important. The state of the art is working with error rates of at most 15% (but usually much les…

Read page 8 [206] of this paper to see the numbers they're providing. The PacBio reads are listed at up to 15% error rate. Their experiments were synthetic data from 15-40% error rates.

A few comments:

1. This was published in ASPLOS, not a comp bio journal, which should raise some eyebrows.

2. As another commenter said, error correction requires a reference, either by alignment, kmer filtering, or consensus. However, this is usually part of any assembly pipeline.

3. Like you said, they failed to compare to actual state of the art alignment tools. They do cite Canu, but they don't use it as their baseline. The issue here is that the current best long read assemblers use locality-sensitive hashing. The Canu paper even states that miniasm can produce a higher-quality assembly for CHM1 in 1/400 the CPU time (though their discussion of the table selectively ignores this fact). Considering that a combination of algorithm and implementation is able to provide a 400x speedup makes this result seem significantly less impressive, and that a comparison of CPU time is slightly unfair to miniasm considering how perfectly it threadscales, which would make the other tools look even worse by comparison. Additionally, for problems where you really need the improvement in performance, the GPU RAM limitations and communication overhead would likely incur significant penalties to their method. It's not simply a matter of how many reads a second you can process because it's a quadratic overlap problem.

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#39

Post by klmr »

stochastic_monk wrote: post #38
klmr wrote: post #27

> The long read technology comes with a drawback though – high error rates in sequencing of between 15%-40% These error rate estimates are seriously outdated (the quoted number is from a 2015 paper but it was obsolete pretty much as soon as that paper was published). Long-read technologies are evolving rapidly, so this is important. The state of the art is working with error rates of at most 15% (but usually much les…

Read page 8 [206] of this paper to see the numbers they're providing. The PacBio reads are listed at up to 15% error rate. Their experiments were synthetic data from 15-40% error rates. A few comments: 1. This was published in ASPLOS, not a comp bio journal, which should raise some eyebrows. 2. As another commenter said, error correction requires a reference, either by alignment, kmer filtering, or consensus. However…

Their 40% number comes from the paper I referenced (i.e. that’s the one the Darwin paper references). I completely agree with your point 2. I’m not sure I agree with your first point: I think it’s pretty reasonable for it to have been published in that venue.

Re: Darwin: a genomics co-processor provides up to 15,000x acceleration

#40

Post by klmr »

searine wrote: post #37
dekhn wrote: post #36

There have been a lot of co-processors in the genomics and sequencing world. They have all failed for obvious business reasons. It's not even clear that any genomics problem that exists today can't be solved with conventional hardware. Most genomics programs reach a few percent of the capacity of the hardware, so I think more software tuning and design is the right approach.

I agree. The problem in genomics isn't how fast alignment is going, it is where to put the gobs and gobs of data. The bottleneck in speed is how fast you can flip a DNA sequencer, which right now with an illumina Novaseq that bottleneck is a few days per run. That gives plenty of time to process each runs data on an HPC on current software for alignment and snp calling. However, when each runfolder takes up 30Tb and…

That’s why compression of sequencing data is becoming so important. CRAM is nowhere near the theoretical limit (CRAM achieves ~2x over BAM; commercial products currently achieve up to 6x).

(COI disclaimer: I work for such a company.)

Post reply on HN