A Desperate Plea for a Free Software Alternative to Aspera (2018)

Post by CaliforniaKarl » Mon, Dec 30, 2019, 4:09 AM UTC

reidacdc wrote: post #95

fanf2 wrote: post #5

Earlier quoted context omitted.

Yes, I got a similar list in response to my link on Twitter https://twitter.com/fanf/status/1210858696558944256 But it looks like GridFTP and Globus are basically dead https://opensciencegrid.org/technology/policy/gridftp-gsi-mi... Tsunami sounds good but maybe needs some updating? I haven’t looked closely...

Globus is definitely not dead[1], it has an active community with strong uptake in the Materials Science world (where I work), among others. The Globus folks have a repo for many popular Linux distros, it's reasonably straightforward to get started. (I'm a user of it, not a developer or necessarily an advocate.) It's not free for "managed" (Enterprise) applications, though, which is what the OP seems to be looking fo…

Post by bsder » Mon, Dec 30, 2019, 4:23 AM UTC

ncmncm wrote: post #44

Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…

Post by zxcmx » Mon, Dec 30, 2019, 4:29 AM UTC

ncmncm wrote: post #44

Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…

Post by [deleted] » Mon, Dec 30, 2019, 4:39 AM UTC

MrEldritch wrote: post #98

grizzles wrote: post #97

Earlier quoted context omitted.

This is something bittorrent could get better at. It would be cool to update the bittorent spec to use eg. a resnet to predict endpoint and path saturation from moment to moment. It's kind of amazing really that the near future will probably be about removing massive amounts of code & design from all sorts of incredibly carefully engineered systems like the linux kernel in favor of a pile of linear algebra that can j…

If you think the best answer to predicting things is to just throw a neural net at it, you should probably learn more control theory. :) (Less flippantly - neural nets are mainly for the case where you can't effectively do manual feature engineering because your inputs are too varied and complex, you don't particularly care about the mathematical properties or guarantees of the solution, and where you don't even real…

Post by branzo » Mon, Dec 30, 2019, 5:02 AM UTC

YarickR2 wrote: post #25

ovidiul wrote: post #17

This looks interesting as well as it's developed at CERN and uses plain tcp protocol https://github.com/fast-data-transfer/fdt

Second this. When we needed to deal with somewhat similar issues (quickly transfer multi-gigabyte archives over massively unreliable link) , we ended up with FDT ; it can be easily scripted , and transfers data between two hosts (without using a network of torrent peers) at that hosts' link speed. Used it to transfer builds between US and Russia, and Russia and China .