A proprietary protocol that somehow facilities throughout orders of magnitude than non proprietary alternatives. I call BS.
A comment that calls BS but has no facts, I call BS?
A Desperate Plea for a Free Software Alternative to Aspera (2018)
161–170 of 172 posts
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#162Try NKN (New Kind of Network) for this task
https://medium.com/nknetwork/nkn-file-transfer-high-throughp...
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#163Another commercial alternative is M-Strean - non proprietary TCP based parallel streaming.
Limited to work with CIFS / SMB / NFS and S3 compatible / OpenStack Swift compatible / Ceph / Azure storage.
Commercial model is server based with no additional bandwidth charges .
Data stored in a non proprietary fashion so bi-modal access to data possible.
Not sure if it is working with QUIC or not as it is mentioned in blog posts.
Seems heavily focused on the media and entertainment industry.
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#164Question at the meta-level: what software/platforms are producing, analyzing, modifying & consuming this data? Is it possible some legacy systems that produce and consume these massive files on-site would more sensibly run in the cloud, directly & selectively accessing the data chunks they need over fast backbone connections? Also, is there any room for an rsync type approach, sending compressed deltas rather than na…
The systems that produce massive DNA sequence files are machines reading the sequences of actual physical DNA molecules. The cloud can't sequence DNA, as far as I know.
> The cloud can't sequence DNA, as far as I know.
Not yet. Famous last words?
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#165Earlier quoted context omitted.
Nitpicking a bit, but a format that is still used ~90% of the time is not obsolete. Not optimal (anymore) is what you meant, I think.
I mean there is no objective reason to use gzip (deflate) compression anymore, unless whoever has to use the data cannot themselves upgrade and is stuck with a gzip deflator. It is little different from building with ARMs instead of Z80s. Where Z80s were once a good choice, now they are almost always a bad choice. Obsolete. USB sticks vs floppies, OLED vs fluorescent-backed LCD, valacyclovir vs acyclovir, Google vs A…
*... with a gzip inflater.
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#166Earlier quoted context omitted.
Most DNA sequences are only slightly different from sequences you already have. There ought to be mature differs by now that would enable you to send just differences. If not, there is a huge opportunity for somebody.
There are such differs. https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...
Thank you. I see that the state of the art remains very primitive, 18kx results notwithstanding.
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#167Earlier quoted context omitted.
The systems that produce massive DNA sequence files are machines reading the sequences of actual physical DNA molecules. The cloud can't sequence DNA, as far as I know.
Most DNA sequences are only slightly different from sequences you already have. There ought to be mature differs by now that would enable you to send just differences. If not, there is a huge opportunity for somebody.
Sure, you can compress the sequences by diffing against the published genome sequence, and many tools exist to do just that. But that alone maxes out at 50% compression, because the other half of the file is the quality scores that quantify the confidence level of each base call, for which there is no reference. Of course there are patterns in the quality scores, and you can compress these as well using the usual techniques for general compression, but it's going to limit how much you can compress without information loss. In addition, the sequencer assigns an unique ID to each sequence read. It's often ok to discard this information, but occasionally you need it in order to trace a specific problematic sequence through the entire analysis pipeline. So if you want lossless compression, you have to compress these as well. They should be rather compressible, since they are typically a prefix global to the file followed by a number that increments by 1 for each sequence. But for a general compressor you can't just automatically assume that, because according to the (lack of a) file format spec, each ID could be anything as long as they're unique within the file. So if you want guaranteed lossless compression of any FASTQ file, your compressor needs to be sufficiently general.
Beyond that, compression of sequences by encoding only the differences from a specific reference sequence somewhat ties you down to using a specific reference, since that reference sequence is required for decompression. This is inconvenient because the reference sequence for a species is updated over time, and if you want to use the new reference, you'll need to decompress with the old reference and then re-compress with the new one. Do you want to do that for all your data, every time the reference is updated? Reference-based compression also means the compressed files are no longer self-contained, which may not be acceptable for certain use cases.
None of these issues are fundamentally impossible to address, but the point is that it's not nearly as simple as it sounds, and anything fancy you do to try and compress a sequence file will generally impose some additional requirements on what the receiver needs to do in order to read the file. And many receivers are not very technically inclined researchers whose plates are already full of other things they need to be doing besides figuring out how to decompress your new unfamiliar sequence compression format,
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#168Earlier quoted context omitted.
Most DNA sequences are only slightly different from sequences you already have. There ought to be mature differs by now that would enable you to send just differences. If not, there is a huge opportunity for somebody.
Sure, you can compress the sequences by diffing against the published genome sequence, and many tools exist to do just that. But that alone maxes out at 50% compression, because the other half of the file is the quality scores that quantify the confidence level of each base call, for which there is no reference. Of course there are patterns in the quality scores, and you can compress these as well using the usual tec…
Thank you for this enlightening explanation. I should have guessed that it, like most things, is not as easy as it looks.
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#169There are other wan accelerators out there - like Riverbed. But, I think Aspera uses some specialized compression for genomics data, and genomics compression is not easy. My suggestion would be to talk to the telco that provides your wan. If you are using a direct internet connection my suggestion would be : don't.
Riverbed always stank. Aspera never did in-flight compression when I was there. In-flight compression got even less interesting around 2010 when the long fiber links got faster than customers' (shared) storage systems. Users have good reasons to compress at rest, before starting a transfer.
I want to make clear that I have no information on Riverbed's current products.
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#170Btw attempted to contact you at the email you put on your page. Interested in talking about this project.