51–60 of 172 posts
There are other wan accelerators out there - like Riverbed. But, I think Aspera uses some specialized compression for genomics data, and genomics compression is not easy. My suggestion would be to talk to the telco that provides your wan. If you are using a direct internet connection my suggestion would be : don't.
I've used UDT via UDR for rsync optimization and transfers over high-latency links: https://github.com/LabAdvComp/UDR
Why not use torrents over wireguard? From an engineering perspective I'd personally be hard pressed to come up with a more optimal solution for private big data transfer. If Aspera is as good as the parent suggests it's probably because they are operating a bittorrent-like network of geo-distributed peers. It would be very cheap to emulate that using cloud computing providers, especially if you are discarding the dat…
Aspera uses multiple UDP streams to push data. The special sauce is measuring the packet loss quickly enough to make sure your not over saturating the link Apart from that, its a fairly simple protocol.
It absolutely does not measure packet loss at all. The only response to a dropped packet is to request a re-send.
The protocol is simple but depends on very smart sending rate control. Much of the complexity of the current version is to perform well when the receiver has a slow disk, or the link goes through a satellite.
I benchmarked Aspera against alternatives for getting data from on-prem storage into the cloud (google cloud storage) a year or two ago on a 10Gbps link. Aspera was faster on a single stream, but if you have a lot of files to move around (you usually do) you can just multiplex a bunch of tcp streams to get the same throughput. So like in the example on their page, if you have 20 wget's hitting their ftp and there are…
I have seen this software before. I suppose the speed boost results from using multiple parallel connections, meaning it could make better use of aggregate links and multipath networks? Is there anything else to it?
It does what it says and it’s expensive. There’s a couple of commercial packages like this - two that come to mind are vcinity and Signiant. A solution can be made but every time I look at the market you find very few customers with a real need to move terabytes that can’t solve their problem with a snowball-style solution. Those that do usually also have the economics to justify a commercial package.
Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…
Have you tried https://github.com/openrq-team/OpenRQ or https://github.com/harmony-one/go-raptorq or https://github.com/cberner/raptorq (the last seems to have excellent decoding/encoding performance)? (Raptor codes (a type of FEC) can essentially transfer over UDP at the underlying line rate, even if packet drop is high . In other words, if it is a 1 gbit/s link, with 5% packet loss, you will be able to send over 90…
You don't need 5% packet loss on a high latency 1Gbps link to trash performance. Even .1% packet loss is enough to cut throughput massively on long fat networks (LFNs). Erasure coding helps, but it demands much more CPU than simply having a very wide window for retransmits. For bulk data transfer of files, retransmitting any part of a 1GB window is trivial.
Why not use torrents over wireguard? From an engineering perspective I'd personally be hard pressed to come up with a more optimal solution for private big data transfer. If Aspera is as good as the parent suggests it's probably because they are operating a bittorrent-like network of geo-distributed peers. It would be very cheap to emulate that using cloud computing providers, especially if you are discarding the dat…
You can compress your data beforehand using zip or another tool. Eg. Tools like https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...
Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.
I benchmarked Aspera against alternatives for getting data from on-prem storage into the cloud (google cloud storage) a year or two ago on a 10Gbps link. Aspera was faster on a single stream, but if you have a lot of files to move around (you usually do) you can just multiplex a bunch of tcp streams to get the same throughput. So like in the example on their page, if you have 20 wget's hitting their ftp and there are…
Sending on lots of TCP streams can get you to 80% line utilization if the drop rate is good and low. That's often good enough. On some lines, with the "wrong" number of TCP connections, the rate will oscillate, and you will be lucky to get 50%. Tuning that is a black art some people enjoy.
Looking at the problem holistically - if you're going to spend money on Aspera to fix situations with high packet loss, maybe you're better off spending money on a higher quality connection and using tcp.
Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…
Would you be willing to talk more about this, for someone not as intimately-familiar with the details?