Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Desperate Plea for a Free Software Alternative to Aspera (2018)

ccdatalab.org

Post reply on HN
Print view View original

151–160 of 172 posts

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#151

Post by exikyut »

ncmncm wrote: post #145
exikyut wrote: post #133

Earlier quoted context omitted.

As a general question, I'm curious how a tactile learner might be able to get a decent grasp of pretty much everything you just described, in a practical setting, without breaking the bank. :) How good is GNSS at simulating latency, buffer saturation, etc on a simulated network and how much complexity would I need to contemplate factoring in to build an accurate model of the average organically evolved, ad-hoc, multi…

It really is a matter of learning the hard way the phenomena that show up on real networks in the wild. Aspera spent a decade incorporating responses to all these lessons into their code. You can get 80% there quickly, 90% there in two years, but the rest is just hard. A versatile simulation environment feeding the algorithm with synthetic time events according to a configured schedule is probably the best way to ram…

I see. I get the impression Aspera is built on a really simple solution, "pressure-adjusted" over a very long time to account for the fact that the real world isn't a vacuum :D

Hearing I can get to 80% quickly is encouraging from a MVP viability point of view, thanks. (And also from a fail-fast perspective; I've now gotten to wondering how much additional performance might be eked out of start-stop style traffic, and it's good to know it won't take long to find out whether the pursuit is worthless or not. Yay.)

I read the second half of that second paragraph as describing a network model simulation that "compiles" a particular routing graph/topology into a set-in-stone sequence of packet events that you then later analyze...? (I'm interpreting "the algorithm" is my target application code, and "synthetic time events" and "configured schedule" as hints at non-real-time pregeneration. This may be incorrect.)

Eek, the delay distributions you describe almost sound like they might be NP-complete to solve.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#152

Post by matsur »

phonon wrote: post #12

Have you tried https://github.com/openrq-team/OpenRQ or https://github.com/harmony-one/go-raptorq or https://github.com/cberner/raptorq (the last seems to have excellent decoding/encoding performance)? (Raptor codes (a type of FEC) can essentially transfer over UDP at the underlying line rate, even if packet drop is high . In other words, if it is a 1 gbit/s link, with 5% packet loss, you will be able to send over 90…

I’m on the product team at Cloudflare. If you run a throughput sensitive workload on Aspera or similar, I’d love to chat — rustam@cloudflare.com

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#153

Post by ncmncm »

exikyut wrote: post #131
ncmncm wrote: post #119

Earlier quoted context omitted.

Didn't use to be. VIA/centaur was the first to add AES instructions, later picked up by Intel and AMD. Its not free, but not expensive. A brilliant woman from, where, Slovenia? figured a way to use SSE instructions to encrypt just as fast as the built-in instructions, I think 2 cycles per byte. I guess DJB's ciphers run much the same way, nowadays.

If you ever track down a citation for that I'd love to know more.

Might be Emilia Kasper, now apparently at Google. Probably it was 2 cycles per bit, not byte.

There are pretty good references starting from https://crypto.stanford.edu/vpaes/ .

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#154

Post by ncmncm »

exikyut wrote: post #151
ncmncm wrote: post #145

Earlier quoted context omitted.

It really is a matter of learning the hard way the phenomena that show up on real networks in the wild. Aspera spent a decade incorporating responses to all these lessons into their code. You can get 80% there quickly, 90% there in two years, but the rest is just hard. A versatile simulation environment feeding the algorithm with synthetic time events according to a configured schedule is probably the best way to ram…

I see. I get the impression Aspera is built on a really simple solution, "pressure-adjusted" over a very long time to account for the fact that the real world isn't a vacuum :D Hearing I can get to 80% quickly is encouraging from a MVP viability point of view, thanks. (And also from a fail-fast perspective; I've now gotten to wondering how much additional performance might be eked out of start-stop style traffic, and…

The idea of fast simulation is to feed just your sending-rate update algorithm with fake time samples, with nothing like actual packets anywhere, so you can get very quick evaluation of your algorithm's numerical behavior under a wide variety of conditions.

Routes tend to start monomodal and split and rejoin at specific times, so it is not usually as bad as you imagine. Also, the number of routes does not get large.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#155

Post by ncmncm »

exikyut wrote: post #131
ncmncm wrote: post #119

Earlier quoted context omitted.

Didn't use to be. VIA/centaur was the first to add AES instructions, later picked up by Intel and AMD. Its not free, but not expensive. A brilliant woman from, where, Slovenia? figured a way to use SSE instructions to encrypt just as fast as the built-in instructions, I think 2 cycles per byte. I guess DJB's ciphers run much the same way, nowadays.

If you ever track down a citation for that I'd love to know more.

https://news.ycombinator.com/item?id=13364762 has interesting discussion.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#156

Post by ncmncm »

sgt101 wrote: post #135
ncmncm wrote: post #82

Earlier quoted context omitted.

Yes, but that typically takes government cooperation, and you still don't get better latency -- even if you have neutrinos.

It doesn't, it takes buying access to a link. Latency isn't the problem here - it's throughput.

Oh, how nice it would be to live in such a simple world.

Latency turns out to have a very great deal to do with how hard it is to get the nominal throughput you pay for.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#157

Post by ncmncm »

exikyut wrote: post #148
ncmncm wrote: post #144

Earlier quoted context omitted.

To be clear, there is absolutely no practical means of high-speed data communication using neutrinos. That is not to say there will never be any.

Ah, I see. So the labs you refer to were trying to make this work? I'm curious what sort of performance level they achieved, even if eg just multiples of single or tens of bytes/sec. Thanks for the info.

No, they were just producing neutrinos, detecting them, and measuring transit time.

But you have to turn the neutrino flux off and on, or at least vary the intensity, to have something to measure.

You might have been able to get tens of bits per second with their apparatus. Or per hour.

Look up "superluminal neutrinos" and work your way past the scandal to the actual experiment..

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#158

Post by zxcvbn4038 »

I feel the author’s pain - Aspera is a clunky 90s styled Ruby on Rails app that is about the most cloud unfriendly piece of software I’ve encountered, I’d love an alternative.

I honestly don’t care about about their proprietary UDP protocol, it’s nothing special, just another way to copy bits onto a wire. Dime a dozen.

The true value of Aspera is they provide an integrated browser plug-in that lets the technically challenged reliably upload large files. If the transfer is interrupted or either side changes addresses it deals with it gracefully. It’s also a bridge to AWS S3.

I spent some time looking for a replacement and while there are numerous download managers that facilitate people downloading large files, I couldn’t find any upload managers with the same level of integration and polish.

About the closet thing I could find is Cyberduck, but it’s not integrated with the web browser, not as easy for technically challenged people to use, and there is no support (community support but seems really hit and miss). However it does make good use of Amazon’s multiple upload api and will happily fill whatever wire it’s connected to.

Torrent software has largely the same pros and cons Cyberduck does.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#159

Post by ncmncm »

bogomipz wrote: post #140
ncmncm wrote: post #78

Earlier quoted context omitted.

Doesn't TCP have an ability to re-send dropped packets, now, without sending everything after them? I thought they called it selective re-send, or selective ack, or something. Maybe the receiver could send an ACK with a bitmap of which packets it got, and the sender would fill in the holes?

Yes it has had selective acknowledgments(SACK) for quite a long time(RFC 2018.) I thought the OP was referring to something perhaps more recent since I believe SACK might predate Aspera.

Most of our customers didn't have SACK-capable network stacks. It used to take a very long time for IETF specs to find their way into deployed systems.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#160

Post by ncmncm »

exikyut wrote: post #132
ncmncm wrote: post #121

Earlier quoted context omitted.

Right, they didn't want us encrypting what they were going to encrypt in their routers anyway. "Trusted" is a technical term. So they started using Aspera to send video from the UAVs over Iraq etc. to the Pentagon. Before that, they were literally flying boxes of videotapes (or CDs? Tapes seem hard to believe, except this was govt.) to a warehouse, where analysts could check them out, like from your neighborhood libr…

Sounds like this was a (little?) while ago. On first read I thought you meant Aspera ran on the UAVs :) but I now realize the high unlikeliness of that...

At the time (mid-late 2000s) the video output from the UAVs was sent in the clear, in analog format, and recorded in analog format. So they had to start digitizing it in the field to transfer as files.

Jihadists could tune in an ordinary scanner, and watch the video coming from them. Embarrassing.

Post reply on HN