A Desperate Plea for a Free Software Alternative to Aspera (2018)

Post by etaioinshrdlu » Mon, Dec 30, 2019, 5:15 AM UTC

grizzles wrote: post #75

ncmncm wrote: post #58

Earlier quoted context omitted.

Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast. Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.

Zip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known. The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.

Post by ncmncm » Mon, Dec 30, 2019, 5:42 AM UTC

zxcmx wrote: post #105

ncmncm wrote: post #44

Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…

I'm curious how much the observed speed of Aspera on the Internet might rely on "nicer" protocols (TCP) backing off as Aspera causes congestion^H^H maximises link utilisation. Do you think the protocol might cause problems if it were more widely used?

Post by na85 » Mon, Dec 30, 2019, 5:50 AM UTC

dnautics wrote: post #86

aamargulies wrote: post #49

Try looking at bbcp perhaps? http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm

bbcp doesn't work so well for some reason. Where I used to work we had a senior dev that claimed that he could just repurpose bbcp for this. Then he gave up and wrote something custom in c++. Then he rewrote it in go, but used (among other things) non-multithreadsafe primitives, then when asked to make it encrypted, he used epoll with tls (which is apparently not a thing in go? I don't know). I told him he should jus…

Post by tonyarkles » Mon, Dec 30, 2019, 6:02 AM UTC

grizzles wrote: post #100

MrEldritch wrote: post #98

Earlier quoted context omitted.

If you think the best answer to predicting things is to just throw a neural net at it, you should probably learn more control theory. :) (Less flippantly - neural nets are mainly for the case where you can't effectively do manual feature engineering because your inputs are too varied and complex, you don't particularly care about the mathematical properties or guarantees of the solution, and where you don't even real…

I was thinking in terms of relative human effort. Adding a neural network into an open source project like bittorrent is going to be very fast to integrate and will probably offer performance that meets the satisficing criteria of being instantly better than the currently offered level of performance. Edit: By the way, I accepted your initial opinion about traditional approaches being better at face value but that wa…

Post by ncmncm » Mon, Dec 30, 2019, 6:17 AM UTC

light_hue_1 wrote: post #116

ncmncm wrote: post #102

Earlier quoted context omitted.

I solved it by cryptographically signing each packet. The signature was checked before decrypting the packet. Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks.

> Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks. Wow. I had no idea. This may be one of the dumbest things I've ever read. Encryption is free.