Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Desperate Plea for a Free Software Alternative to Aspera (2018)

ccdatalab.org

Post reply on HN
Print view View original

141–150 of 172 posts

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#141

Post by sgt101 »

exikyut wrote: post #136
ncmncm wrote: post #106

Earlier quoted context omitted.

Certain labs were sending neutrinos right through the earth, gaining a measurable reduction in transit time vs great-circle overland signal propagation. (At first they thought they were gaining even more, by superluminal transit, but that turned out to illusory.) They did not put any data in the neutrino channel, to my knowledge, but in principle they could have. You might think you would need quite a deep chord to b…

If there's any possible way you could share more about this... I am very very interested to learn more.

Bear in mind that you have to put the signal in and get it out of the channel as well as transmit it. I highly doubt that neutrino signaling kit is as optimized for doing this as photonics.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#142

Post by fanf2 »

eps wrote: post #137
ncmncm wrote: post #44

Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…

Rate control sounds nearly identical to TCP Vegas then, no? But out of order delivery and (I'm guessing) out of order retransmits appear to be unique to fasp. What did you mean about these being unique "until recently"? [edit - nevermind, saw below you were referring to sack] PS. I remember looking at fasp ~10 years ago and it looked like a fantastic tech. Someone's taking things that many people only talked about an…

And more recently Google's BBR is a newer delay-controlled congestion control algorithm - http://www.potaroo.net/ispcol/2017-05/bbr.html

So it'll be interesting to see if BBR+QUIC can be made to do what FASP does.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#143

Post by dnautics »

na85 wrote: post #115
dnautics wrote: post #86

Earlier quoted context omitted.

bbcp doesn't work so well for some reason. Where I used to work we had a senior dev that claimed that he could just repurpose bbcp for this. Then he gave up and wrote something custom in c++. Then he rewrote it in go, but used (among other things) non-multithreadsafe primitives, then when asked to make it encrypted, he used epoll with tls (which is apparently not a thing in go? I don't know). I told him he should jus…

>On the other hand, maybe bbcp will work, and the senior dev just didn't know what he was doing I mean... based on your description it sure sounds like he didn't know what he was doing.

Well yeah but the competent dev would have told you that it wouldn't work for OP.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#144

Post by ncmncm »

exikyut wrote: post #136
ncmncm wrote: post #106

Earlier quoted context omitted.

Certain labs were sending neutrinos right through the earth, gaining a measurable reduction in transit time vs great-circle overland signal propagation. (At first they thought they were gaining even more, by superluminal transit, but that turned out to illusory.) They did not put any data in the neutrino channel, to my knowledge, but in principle they could have. You might think you would need quite a deep chord to b…

If there's any possible way you could share more about this... I am very very interested to learn more.

To be clear, there is absolutely no practical means of high-speed data communication using neutrinos.

That is not to say there will never be any.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#145

Post by ncmncm »

exikyut wrote: post #133
ncmncm wrote: post #70

Earlier quoted context omitted.

I've reported most of what I know. I was involved in the data layer. Flow control was done by the PhDs. But, imagine the route as a series of packet queues. Packet is sent, arrives at the first router, spends time in the queue, gets sent, put on the next queue, waits, goes, until it's delivered. If the total time taken, compared to the last packet's, is more, that means the queues are growing. If the total time is le…

As a general question, I'm curious how a tactile learner might be able to get a decent grasp of pretty much everything you just described, in a practical setting, without breaking the bank. :) How good is GNSS at simulating latency, buffer saturation, etc on a simulated network and how much complexity would I need to contemplate factoring in to build an accurate model of the average organically evolved, ad-hoc, multi…

It really is a matter of learning the hard way the phenomena that show up on real networks in the wild. Aspera spent a decade incorporating responses to all these lessons into their code.

You can get 80% there quickly, 90% there in two years, but the rest is just hard. A versatile simulation environment feeding the algorithm with synthetic time events according to a configured schedule is probably the best way to ramp up quickly, but I don't know anything about GNSS.

One tricky phenomenon is bimodal and even N-modal delay samples from traffic split across different routes.

Around 2014, Aspera replaced most of the code wrapped around the algorithms to become much more nimble about applying them in more circumstances.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#146

Post by burntwater »

danbmil99 wrote: post #104

Question at the meta-level: what software/platforms are producing, analyzing, modifying & consuming this data? Is it possible some legacy systems that produce and consume these massive files on-site would more sensibly run in the cloud, directly & selectively accessing the data chunks they need over fast backbone connections? Also, is there any room for an rsync type approach, sending compressed deltas rather than na…

Aspera is the standard in broadcast video for transferring video files. Typically you wouldn't want just a small piece, you would want the entire hour-long, 3 terabyte video.

Also, you would still need to get the data into the cloud in the first place.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#147

Post by flas9sd »

justinsaccount wrote: post #41

The 'proof' shown that aspera is 200x faster shows an ETA of 2:15 compared to 00:09. First of all, this is 15x faster, not 200. I tried downloading the file using axel -n 32, and it took 1:28, which is only 9x slower. Curious though, when it started out with all threads running it was transferring data at 29594.2KB/s and hit 50% after only a few seconds, but by the end with only a few threads left running it was only…

not to take away from the posts plea. Apart from bit vs bytes, it should be noted the remotes are also on different machines with potentially varying network throughput

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#148

Post by exikyut »

ncmncm wrote: post #144
exikyut wrote: post #136

Earlier quoted context omitted.

If there's any possible way you could share more about this... I am very very interested to learn more.

To be clear, there is absolutely no practical means of high-speed data communication using neutrinos. That is not to say there will never be any.

Ah, I see. So the labs you refer to were trying to make this work? I'm curious what sort of performance level they achieved, even if eg just multiples of single or tens of bytes/sec.

Thanks for the info.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#150

Post by ncmncm »

eps wrote: post #137
ncmncm wrote: post #44

Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…

Rate control sounds nearly identical to TCP Vegas then, no? But out of order delivery and (I'm guessing) out of order retransmits appear to be unique to fasp. What did you mean about these being unique "until recently"? [edit - nevermind, saw below you were referring to sack] PS. I remember looking at fasp ~10 years ago and it looked like a fantastic tech. Someone's taking things that many people only talked about an…

No, quite different from Vegas, if the graphs are to be believed. What I would see on real trans-oceanic links was a sub-second ramp up to a flat, stable rate, and stepwise rate changes during the session as other flows started and stopped, with no visible oscillation.

(I am looking at https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/ .)

Fasp does not rely on RTT ACK timing, which is very noisy, but only outgoing delays.

The routers are applying their own algorithms to simulate sane queue timing, but they are doing all kinds of crazy shit under the hood to get better link performance. The queues, IOW, are a fiction provided to the endpoints to simplify their job and generate understandable, thus tunable full-route dynamics.

But the ACKs get completely different treatment, because traditionally nothing cared about ACK timing.

So, "onset of queuing", e.g., is pretty meaningless in practice, and fasp makes no attempt to detect it.

Post reply on HN