gridftp's the tool to use, and Globus makes it easy.
A Desperate Plea for a Free Software Alternative to Aspera (2018)
61–70 of 172 posts
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#62How about mpscp from Sandia?
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#63Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…
Are you familiar with UDT ( http://udt.sourceforge.net/ ) ? How does it compare to Aspera ? Thanks!
I was not familiar with it. It looks like you could build a good-enough file transfer system on top of UDT, but probably not with the rate control protocols they provide.
One problem with UDP protocols over the open internet is that you cannot trust UDP packet contents, which could come from anywhere.
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#64Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…
>"Another advantage, until recently, was out-of-order delivery."
Can you elaborate on this? Which recent development are you referring to? Thanks.
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#65Dat isn't designed for speed, but I think academic data is it's primary goal, so modifying it for speed seems within its charter.
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#66There are other wan accelerators out there - like Riverbed. But, I think Aspera uses some specialized compression for genomics data, and genomics compression is not easy. My suggestion would be to talk to the telco that provides your wan. If you are using a direct internet connection my suggestion would be : don't.
The author is discussing intercontinental data transfers. Not many organizations have a WAN that can cross oceans without hitting the internet.
almost all HE and research institutions participate in nrens which provide 3xacrly those peerings
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#67In the past I see that Tsunami, UDT, or GridFTP were suggested. Any results?
Yes, I got a similar list in response to my link on Twitter https://twitter.com/fanf/status/1210858696558944256 But it looks like GridFTP and Globus are basically dead https://opensciencegrid.org/technology/policy/gridftp-gsi-mi... Tsunami sounds good but maybe needs some updating? I haven’t looked closely...
Globus isn't dead at all globus.org, it just turned into something that solved an actual problem.
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#68Would QUIC or TCP with BBR congestion control help?
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#69Can they not leverage torrents for this purpose?
Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)
#70Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…
> The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of the measurement is less than the transit time. This should make an engineer think "control theory!" (and did). So, the receiver reports a stream of transit time samples back to the sender, which feeds them into a predictor, which controls transmi…
I've reported most of what I know. I was involved in the data layer. Flow control was done by the PhDs.
But, imagine the route as a series of packet queues. Packet is sent, arrives at the first router, spends time in the queue, gets sent, put on the next queue, waits, goes, until it's delivered. If the total time taken, compared to the last packet's, is more, that means the queues are growing. If the total time is less, the queues are shrinking. You adjust your rate to stabilize them. Other flows are being added and removed all the time, so the optimal rate varies in steps over the course of the session.
This involves control theory, Kalman filters, PID, the whole business.