Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Desperate Plea for a Free Software Alternative to Aspera (2018)

ccdatalab.org

Post reply on HN
Print view View original

21–30 of 172 posts

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#21

Post by big_chungus »

sansnomme wrote: post #19
dboreham wrote: post #10

A proprietary protocol that somehow facilities throughout orders of magnitude than non proprietary alternatives. I call BS.

If they are like Riot and have their own fiber, it is not unlikely. There are tons of boring optimizations open source software also won't bother with without significant sponsorship (some could be specific to a particular file format). If you look at most network (and also video) protocols, they are accumulations of decades of improvements rather than neural network style overnight breakthroughs.

Do you know of any resources where I can learn more about these boring optimizations?

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#22

Post by quasarj »

My group is about to start using Aspera as well.. I too wish there was a good alternative I could suggest to my bosses. And I'm still skeptical of their claims... but I haven't gotten to test it over an actual bad link yet (over a good link it does nothing, which a simple test will prove).

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#23

Post by saagarjha »

> Aspera is owned by IBM - does their client spy on what else is running on my system and report it back to their headquarters as business intelligence? If asked, would they share this information with the government? Without the source code, it’s impossible to tell!

I mean, with some reverse engineering, mentioned later in the post, it’s really not.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#24

Post by SaxonRobber »

big_chungus wrote: post #21
sansnomme wrote: post #19

Earlier quoted context omitted.

If they are like Riot and have their own fiber, it is not unlikely. There are tons of boring optimizations open source software also won't bother with without significant sponsorship (some could be specific to a particular file format). If you look at most network (and also video) protocols, they are accumulations of decades of improvements rather than neural network style overnight breakthroughs.

Do you know of any resources where I can learn more about these boring optimizations?

MIT open courseware has a course on Performance Engineering which details some of the types of low level optimizations that can be done. It requires a lot of measuring and problem specific solutions.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#25

Post by YarickR2 »

ovidiul wrote: post #17

This looks interesting as well as it's developed at CERN and uses plain tcp protocol https://github.com/fast-data-transfer/fdt

Second this. When we needed to deal with somewhat similar issues (quickly transfer multi-gigabyte archives over massively unreliable link) , we ended up with FDT ; it can be easily scripted , and transfers data between two hosts (without using a network of torrent peers) at that hosts' link speed. Used it to transfer builds between US and Russia, and Russia and China .

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#26

Post by brians »

dboreham wrote: post #10

A proprietary protocol that somehow facilities throughout orders of magnitude than non proprietary alternatives. I call BS.

I know about a couple of these. Some use very specific compression based on external knowledge—like that this is all office xml documents, with freedom to study the set of the last year’s documents. Some work by eliciting pessimal behavior from TCP flows sharing the same link—they’re really bad neighbors!

Some use a family of different approaches to cope with different performance domains. Some use intermediate relays to cut the bandwidth-deat product by reducing delay.

But the big key is this: standard tools aren’t even trying to be good at this. They’re aimed at making an Internet work, not at optimizing and particular flow.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#27

Post by brians »

SaxonRobber wrote: post #24
big_chungus wrote: post #21

Earlier quoted context omitted.

Do you know of any resources where I can learn more about these boring optimizations?

MIT open courseware has a course on Performance Engineering which details some of the types of low level optimizations that can be done. It requires a lot of measuring and problem specific solutions.

I’ve helped with that course in the past. It’s awesome. It’s completely unrelated to WAN networking. I found the IETF groups working on these problems more helpful.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#30

Post by grizzles »

grizzles wrote: post #18

Why not use torrents over wireguard? From an engineering perspective I'd personally be hard pressed to come up with a more optimal solution for private big data transfer. If Aspera is as good as the parent suggests it's probably because they are operating a bittorrent-like network of geo-distributed peers. It would be very cheap to emulate that using cloud computing providers, especially if you are discarding the dat…

You can compress your data beforehand using zip or another tool. Eg. Tools like https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...
Post reply on HN