Live data from Hacker News

A Desperate Plea for a Free Software Alternative to Aspera (2018)

ccdatalab.org

171–172 of 172 posts

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#171

Earlier quoted context omitted.

Zip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known. The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.

I thought ZIP files only supported a bunch of old algorithms like deflate. Maybe you can make an exotic zip file with zstd but doubt there are any tools that will understand it.

Version 6.3.3 from the Library of Congress - https://www.loc.gov/preservation/digital/formats/digformatsp...

Writeup on zip file Appnotes: https://entropymine.wordpress.com/2019/08/22/survey-of-zip-a...

Actual PKWare AppNotes: https://support.pkware.com/display/PKZIP/Application+Note+Ar...

Section 4.4.5 is "compression method"

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#172
Beyond more profound changes to the underlying transport protocol, bandwidth acceleration can be provide through a combination with fast compression.

A good setup here would be to pipe data to `tar`, then `zstd`, then `ssh`/`scp`, and do the reverse on the receiver side.

From an end-user standpoint, all this could be abstracted away if one could get `wget` (named in the article) to support zstd compression natively, which is possible since zstd is also a web compression standard.

Such compression support is, by the way, available in `wget2`.

Post reply on HN