91–100 of 172 posts
It's unfortunate how many researchers don't do such research...
Try looking at bbcp perhaps? http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm
bbcp doesn't work so well for some reason. Where I used to work we had a senior dev that claimed that he could just repurpose bbcp for this. Then he gave up and wrote something custom in c++. Then he rewrote it in go, but used (among other things) non-multithreadsafe primitives, then when asked to make it encrypted, he used epoll with tls (which is apparently not a thing in go? I don't know). I told him he should jus…
There's a meta answer about the general approach to such problems: Ask what other research areas must have tackled it, and then find out what they do. The obvious answer in this case is HEP, which would lead to some of the tools mentioned. (There's also https://www.psc.edu/index.php/hpn-ssh but I don't know how it compares.) It's unfortunate how many researchers don't do such research...
In the past I see that Tsunami, UDT, or GridFTP were suggested. Any results?
Yes, I got a similar list in response to my link on Twitter https://twitter.com/fanf/status/1210858696558944256 But it looks like GridFTP and Globus are basically dead https://opensciencegrid.org/technology/policy/gridftp-gsi-mi... Tsunami sounds good but maybe needs some updating? I haven’t looked closely...
It's not free for "managed" (Enterprise) applications, though, which is what the OP seems to be looking for, and I'm not sure it's the best choice for high-speed.
Something that I think is free is the CERN FTS [2], which can use a GridFTP back-end, so possibly you can roll your own high-speed big-data infrastructure that way.
My group is about to start using Aspera as well.. I too wish there was a good alternative I could suggest to my bosses. And I'm still skeptical of their claims... but I haven't gotten to test it over an actual bad link yet (over a good link it does nothing, which a simple test will prove).
It's being run out of UChicago, and I know they have at least one large company using them. Feel free to reach out to the Globus team, or shoot me an email!
Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…
It's kind of amazing really that the near future will probably be about removing massive amounts of code & design from all sorts of incredibly carefully engineered systems like the linux kernel in favor of a pile of linear algebra that can just figure things out. resistance is futile.
Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…
This is something bittorrent could get better at. It would be cool to update the bittorent spec to use eg. a resnet to predict endpoint and path saturation from moment to moment. It's kind of amazing really that the near future will probably be about removing massive amounts of code & design from all sorts of incredibly carefully engineered systems like the linux kernel in favor of a pile of linear algebra that can j…
(Less flippantly - neural nets are mainly for the case where you can't effectively do manual feature engineering because your inputs are too varied and complex, you don't particularly care about the mathematical properties or guarantees of the solution, and where you don't even really know what the solution should look like [so are forced to randomly initialize a huge nonlinear system and optimize it until useful behaviors pop out]. Something like predicting path saturation would be far better suited to traditional approaches.)
In the past I see that Tsunami, UDT, or GridFTP were suggested. Any results?
I worked with a couple of the commercial alternatives to aspera and aspera as well and I used UDT, Tsunami, GridFTP, syncthing[1] as a poor mans alternative to Aspera. For real transfer with big (> 500TB) Aspera will deliver what it says over WAN (over the Atlantic). If you have better connections and not so much data syncthing will probably work if you can have someone manually take care of all exceptions.[2] If you…
Earlier quoted context omitted.
This is something bittorrent could get better at. It would be cool to update the bittorent spec to use eg. a resnet to predict endpoint and path saturation from moment to moment. It's kind of amazing really that the near future will probably be about removing massive amounts of code & design from all sorts of incredibly carefully engineered systems like the linux kernel in favor of a pile of linear algebra that can j…
If you think the best answer to predicting things is to just throw a neural net at it, you should probably learn more control theory. :) (Less flippantly - neural nets are mainly for the case where you can't effectively do manual feature engineering because your inputs are too varied and complex, you don't particularly care about the mathematical properties or guarantees of the solution, and where you don't even real…
Edit: By the way, I accepted your initial opinion about traditional approaches being better at face value but that was like ~45 minutes ago. After thinking about it a bit more I realized that there is alot of interesting network data that could be fed into such a NN, moving it (ding!) back into the nonlinear/complex system column. Suddenly my approach is looking competitive to also being the optimal one (excluding hybrids).