Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Desperate Plea for a Free Software Alternative to Aspera (2018)

ccdatalab.org

Post reply on HN
Print view View original

71–80 of 172 posts

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#71

Post by phonon »

Scaevolus wrote: post #46
phonon wrote: post #12

Have you tried https://github.com/openrq-team/OpenRQ or https://github.com/harmony-one/go-raptorq or https://github.com/cberner/raptorq (the last seems to have excellent decoding/encoding performance)? (Raptor codes (a type of FEC) can essentially transfer over UDP at the underlying line rate, even if packet drop is high . In other words, if it is a 1 gbit/s link, with 5% packet loss, you will be able to send over 90…

You don't need 5% packet loss on a high latency 1Gbps link to trash performance. Even .1% packet loss is enough to cut throughput massively on long fat networks (LFNs). Erasure coding helps, but it demands much more CPU than simply having a very wide window for retransmits. For bulk data transfer of files, retransmitting any part of a 1GB window is trivial.

Yes, of course--even a low rate of packet loss where the bandwidth delay product is high will cause you to end up with throughout dropping to critically terrible levels. Tweaking TCP can help, but a high performance RaptorQ system can work at packet loss levels that TCP could never handle.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#72

Post by phonon »

traverseda wrote: post #57
Scaevolus wrote: post #46

Earlier quoted context omitted.

You don't need 5% packet loss on a high latency 1Gbps link to trash performance. Even .1% packet loss is enough to cut throughput massively on long fat networks (LFNs). Erasure coding helps, but it demands much more CPU than simply having a very wide window for retransmits. For bulk data transfer of files, retransmitting any part of a 1GB window is trivial.

You mean with raptorQ 5% loss trashes performance by more than 5%?

No, the overhead is fairly minimal. A 5% packet loss (which is very bad) on a gigabit link where the sender is thousands of miles away might have an effective throughput of 930 mbits per second with RaptorQ. With regular TCP you would get only a few megabits/s, as it would back down until the loss rate was nearly zero.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#74

Post by ncmncm »

dboreham wrote: post #10

A proprietary protocol that somehow facilities throughout orders of magnitude than non proprietary alternatives. I call BS.

Orders of magnitude improvement over TCP is easy when the bandwidth-delay product and loss rate keep TCP stuck at 0.1% of line rate. It says more about how bad TCP is on those lines than about how good the alternative is.

But the alternative is good, too. Worth the price? Depends on non-technical factors.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#75

Post by grizzles »

ncmncm wrote: post #58
grizzles wrote: post #30

Earlier quoted context omitted.

You can compress your data beforehand using zip or another tool. Eg. Tools like https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...

Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast. Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.

Zip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known.

The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#77

Post by StavrosK »

justinsaccount wrote: post #41

The 'proof' shown that aspera is 200x faster shows an ETA of 2:15 compared to 00:09. First of all, this is 15x faster, not 200. I tried downloading the file using axel -n 32, and it took 1:28, which is only 9x slower. Curious though, when it started out with all threads running it was transferring data at 29594.2KB/s and hit 50% after only a few seconds, but by the end with only a few threads left running it was only…

Yeah, and one window shows speeds in MB/s and the other in Mbps (a 10x difference in units right there).

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#78

Post by ncmncm »

bogomipz wrote: post #64
ncmncm wrote: post #44

Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…

>"Another advantage, until recently, was out-of-order delivery." Can you elaborate on this? Which recent development are you referring to? Thanks.

Doesn't TCP have an ability to re-send dropped packets, now, without sending everything after them? I thought they called it selective re-send, or selective ack, or something. Maybe the receiver could send an ACK with a bitmap of which packets it got, and the sender would fill in the holes?

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#79

Post by ncmncm »

grizzles wrote: post #75
ncmncm wrote: post #58

Earlier quoted context omitted.

Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast. Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.

Zip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known. The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.

OK, I was really comparing to zstd and lz4 to gzip, which was the original algorithm in zip, and the one most people still use. Clearly, if you can plug better algorithms into Zip, you should, providing the Zip archive format is still good enough.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#80

Post by justinsaccount »

StavrosK wrote: post #77
justinsaccount wrote: post #41

The 'proof' shown that aspera is 200x faster shows an ETA of 2:15 compared to 00:09. First of all, this is 15x faster, not 200. I tried downloading the file using axel -n 32, and it took 1:28, which is only 9x slower. Curious though, when it started out with all threads running it was transferring data at 29594.2KB/s and hit 50% after only a few seconds, but by the end with only a few threads left running it was only…

Yeah, and one window shows speeds in MB/s and the other in Mbps (a 10x difference in units right there).

ah, I bet they missed that which is how they ended up with 200x instead of under 20x
Post reply on HN