Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Desperate Plea for a Free Software Alternative to Aspera (2018)

ccdatalab.org

Post reply on HN
Print view View original

1–10 of 172 posts

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#5

Post by fanf2 »

TNick wrote: post #3

In the past I see that Tsunami, UDT, or GridFTP were suggested. Any results?

Yes, I got a similar list in response to my link on Twitter https://twitter.com/fanf/status/1210858696558944256

But it looks like GridFTP and Globus are basically dead https://opensciencegrid.org/technology/policy/gridftp-gsi-mi...

Tsunami sounds good but maybe needs some updating? I haven’t looked closely...

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#7

Post by jonke »

TNick wrote: post #3

In the past I see that Tsunami, UDT, or GridFTP were suggested. Any results?

I worked with a couple of the commercial alternatives to aspera and aspera as well and I used UDT, Tsunami, GridFTP, syncthing[1] as a poor mans alternative to Aspera.

For real transfer with big (> 500TB) Aspera will deliver what it says over WAN (over the Atlantic).

If you have better connections and not so much data syncthing will probably work if you can have someone manually take care of all exceptions.[2]

If you know your data and you can build quite a lot of stuff yourself you can get almost the same speed as Aspera with UDT.

There is also a go implementation of UDTs used by kcptun[3] but I haven't tried that.

1 https://github.com/syncthing/syncthing

2 I don't intend to disrespect syncthing here, but when your dealing with TB scale 24/7 things are never really up all the time, the network is unreliable, the sender host filesystem is corrupt and the receiver host filesystem is full or broken or ...

3 https://github.com/xtaci/kcptun

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#8

Post by kylek »

dgemm wrote: post #6

I have seen this software before. I suppose the speed boost results from using multiple parallel connections, meaning it could make better use of aggregate links and multipath networks? Is there anything else to it?

Used a few of their products at a previous company. That's the gist of the protocol. It's UDP-based and the software has intelligence to use bandwidth as efficiently as possible between links. We had a few servers sending to many endpoints concurrently, and I'd regularly see the systems cap out their 10gbit connections.

License costs are pretty brutal and it does need decent amount CPU to make the most of it.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#9

Post by mattrp »

dgemm wrote: post #6

I have seen this software before. I suppose the speed boost results from using multiple parallel connections, meaning it could make better use of aggregate links and multipath networks? Is there anything else to it?

It does what it says and it’s expensive. There’s a couple of commercial packages like this - two that come to mind are vcinity and Signiant. A solution can be made but every time I look at the market you find very few customers with a real need to move terabytes that can’t solve their problem with a snowball-style solution. Those that do usually also have the economics to justify a commercial package.
Post reply on HN