A Desperate Plea for a Free Software Alternative to Aspera (2018)

Its only faster over large high latency links. Instead of using a single TCP stream to push data (with that exponential roll off) it has a TCP session to do accounting, and a bunch of UDP ports to stream the data.

for links around 16ms latency, its not that much faster.

For a 10 gig link with a latency of 100ms, it runs pretty much at wire speed. (assuming you have a machine that can handle it.)

1. Measure the existing performance on specific data sets.

2. Understand how many of the bottlenecks come from the network latency vs. I/O limits vs. CPU bottlenecks (if using compression).

3. See if any domain-specific compression is needed.

4. Document the typical use cases. Understand why the current solution sucks (e.g. requires redundant user actions). Write down user interaction scenarios. Design the UI to be as efficient as possible for those scenarios.

This is a non-trivial amount of work that would require a lot of back-and-forth interaction and on-the-go requirement changes and I don't think it's entirely honest to ask someone to do this work for free in the name of cancer research (after all, you are not donating most of your paycheck to charities, are you?). If the existing solution by IBM sucks, how about making a Request For Proposal [0] and seeing if smaller software vendors could offer something better given that you are actually willing to pay for the work?

P.S. A student/hobbyist can probably whip out some sort of a parallel TCP-like thing with a large window for free, but you would get the same performance by just cranking up the TCP window size via sysctl and using multiple HTTP threads (htcat was suggested earlier in the comments).

[0] https://en.wikipedia.org/wiki/Request_for_proposal

The special sauce is measuring the packet loss quickly enough to make sure your not over saturating the link

Apart from that, its a fairly simple protocol.