Cool. But I think the current state of the art is using minhashes (minimap2) to avoid doing expensive all vs all alignments. There is still a place for more sensitive alignments though.
Darwin: a genomics co-processor provides up to 15,000x acceleration
41–50 of 51 posts
Re: Darwin: a genomics co-processor provides up to 15,000x acceleration
#42Serious question: if the processing problems are classified as NP, then couldn't that processing be outsourced to a blockchain? Why have miners solve for useless problems that have no long lasting impact like "number of zeros in a SHA string" instead of processing data such as this? I can easily see it being the case that the mining problem in a given blockchain could be based on real scientific problems that needs s…
There are droves of the distributed computational projects, mostly based on Boinc [1]. There are projects whose goal is biochemistry applications too [2].
Re: Darwin: a genomics co-processor provides up to 15,000x acceleration
#43Earlier quoted context omitted.
One argument is that the blockchain problem coincides with a "real scientific problem" that needs solving and if the scientific problem leads to some application, which can yield a monetary reward (may be big "if"), this would make it cheaper to launch a 51% attack against this blockchain. The attacker could subsidize their attack by being paid to do science.
Not so serious answer: Hurray for that? :D Serious answer: Things would obviously have to be designed differently. This isn't about distribution of "economic power to the hands of the people" or some other Satoshi notion. The thought I have is more along the lines of Folding@Home http://folding.stanford.edu/
If this isn't about some "Satoshi notion", then what does blockchain have to do with it? Just do something like Folding@Home. There are several projects like that.
Re: Darwin: a genomics co-processor provides up to 15,000x acceleration
#44There have been a lot of co-processors in the genomics and sequencing world. They have all failed for obvious business reasons. It's not even clear that any genomics problem that exists today can't be solved with conventional hardware. Most genomics programs reach a few percent of the capacity of the hardware, so I think more software tuning and design is the right approach.
Oddly enough, I found a similar sentiment echoed last night when I was looking up how widespread the use of ASICs was in various scientific fields. I had discovered an FPGA startup called Edico Genome [1] that seemed to offer substantial advantages in speed of processing, and after searching through reddit came across a post where someone noted that there has been a precedent for FPGA/ASIC related companies to go under in the genomics sphere [2]. The poster there attributed this to an unwillingness to use tools that weren't widely cited, however.
[1] http://edicogenome.com/ [2] https://www.reddit.com/r/bioinformatics/comments/73wcd3/fail...
Re: Darwin: a genomics co-processor provides up to 15,000x acceleration
#45“Up to x% faster” another marketing term that used to work.
Re: Darwin: a genomics co-processor provides up to 15,000x acceleration
#46Earlier quoted context omitted.
I agree. The problem in genomics isn't how fast alignment is going, it is where to put the gobs and gobs of data. The bottleneck in speed is how fast you can flip a DNA sequencer, which right now with an illumina Novaseq that bottleneck is a few days per run. That gives plenty of time to process each runs data on an HPC on current software for alignment and snp calling. However, when each runfolder takes up 30Tb and…
That’s why compression of sequencing data is becoming so important. CRAM is nowhere near the theoretical limit (CRAM achieves ~2x over BAM; commercial products currently achieve up to 6x). (COI disclaimer: I work for such a company.)
No need for the disclaimer, post a link to your benchmarking results.
We don't compress as well as CRAM, but distributed reads across an Apache Spark cluster are much more efficient via Parquet
Re: Darwin: a genomics co-processor provides up to 15,000x acceleration
#47Earlier quoted context omitted.
I agree. The problem in genomics isn't how fast alignment is going, it is where to put the gobs and gobs of data. The bottleneck in speed is how fast you can flip a DNA sequencer, which right now with an illumina Novaseq that bottleneck is a few days per run. That gives plenty of time to process each runs data on an HPC on current software for alignment and snp calling. However, when each runfolder takes up 30Tb and…
That’s why compression of sequencing data is becoming so important. CRAM is nowhere near the theoretical limit (CRAM achieves ~2x over BAM; commercial products currently achieve up to 6x). (COI disclaimer: I work for such a company.)
It's not clear you want to store your sequence data maximally compressed unless you have it archived. If you're doing live work on the data, it's better to have it modestly compressed with a fast decompressor, and a good index so you can minimize your total access.
Re: Darwin: a genomics co-processor provides up to 15,000x acceleration
#48Earlier quoted context omitted.
That’s why compression of sequencing data is becoming so important. CRAM is nowhere near the theoretical limit (CRAM achieves ~2x over BAM; commercial products currently achieve up to 6x). (COI disclaimer: I work for such a company.)
It's not clear you want to store your sequence data maximally compressed unless you have it archived. If you're doing live work on the data, it's better to have it modestly compressed with a fast decompressor, and a good index so you can minimize your total access.
That’s plausible but it turns out not to be true. Disk read latency is larger than the overhead of our decompression, even on fast storage media. As a consequence, better compression actually leads to (slight) performance improvements. Additionally, genomic data is often accessed via relatively slow network storage on clusters or, worse, via the internet. Increasing throughput trumps all other considerations here. You’re right concerning random access and indexing, of course.
Re: Darwin: a genomics co-processor provides up to 15,000x acceleration
#49Earlier quoted context omitted.
It's not clear you want to store your sequence data maximally compressed unless you have it archived. If you're doing live work on the data, it's better to have it modestly compressed with a fast decompressor, and a good index so you can minimize your total access.
That’s plausible but it turns out not to be true. Disk read latency is larger than the overhead of our decompression, even on fast storage media. As a consequence, better compression actually leads to (slight) performance improvements. Additionally, genomic data is often accessed via relatively slow network storage on clusters or, worse, via the internet. Increasing throughput trumps all other considerations here. Yo…
I counter your argument: I have constructed systems that did this. Disk read latency doesn't matter for streaming reads.
The actual math for this for a product at scale is interesting; I built such a product and did the math, and we found that moderate compression gave higher rates. However, this is based on a production-class infrastructure.
Re: Darwin: a genomics co-processor provides up to 15,000x acceleration
#50There have been a lot of co-processors in the genomics and sequencing world. They have all failed for obvious business reasons. It's not even clear that any genomics problem that exists today can't be solved with conventional hardware. Most genomics programs reach a few percent of the capacity of the hardware, so I think more software tuning and design is the right approach.
Oddly enough, I found a similar sentiment echoed last night when I was looking up how widespread the use of ASICs was in various scientific fields. I had discovered an FPGA startup called Edico Genome [1] that seemed to offer substantial advantages in speed of processing, and after searching through reddit came across a post where someone noted that there has been a precedent for FPGA/ASIC related companies to go und…
To deal with the citation problem, many FPGAs and ASICs that implemented BLAST guaranteed "bit-identical output". You could easily verify anything found by the accelerator using BLAST, just more slowly.