11–20 of 172 posts
If you think about giving zmq a go, take a look at my library, here [1], [2]. Maybe we can talk about compiling a test to see how it compares? [1] https://pypi.org/project/p2p0mq/ [2] https://p2p0mq.readthedocs.io/en/latest/
(Raptor codes (a type of FEC) can essentially transfer over UDP at the underlying line rate, even if packet drop is high. In other words, if it is a 1 gbit/s link, with 5% packet loss, you will be able to send over 900 mbits/s without needing any TCP features). This is also helps high latency scenarios, as you don't need test/back off how much data to send...you just send as much Raptor encoded data as you want down the pipe, and as long as you send enough recovery packets with it, you will be able to reconstruct the original flawlessly).
(Also see https://par.nsf.gov/servlets/purl/10066600
http://www1.icsi.berkeley.edu/~pooja/HowUseRaptorQ.pdf
https://a1f9fb7d-b120-4c73-98e3-5cdb4ec8a2ab.filesusr.com/ug...
I have seen this software before. I suppose the speed boost results from using multiple parallel connections, meaning it could make better use of aggregate links and multipath networks? Is there anything else to it?
Used a few of their products at a previous company. That's the gist of the protocol. It's UDP-based and the software has intelligence to use bandwidth as efficiently as possible between links. We had a few servers sending to many endpoints concurrently, and I'd regularly see the systems cap out their 10gbit connections. License costs are pretty brutal and it does need decent amount CPU to make the most of it.
Earlier quoted context omitted.
Used a few of their products at a previous company. That's the gist of the protocol. It's UDP-based and the software has intelligence to use bandwidth as efficiently as possible between links. We had a few servers sending to many endpoints concurrently, and I'd regularly see the systems cap out their 10gbit connections. License costs are pretty brutal and it does need decent amount CPU to make the most of it.
Are you able to recommend a resource for how to use bandwidth most efficiently like this? I'm interested in learning more, and would appreciate it. Also, might using raw sockets not offer better performance and lower CPU usage?
Earlier quoted context omitted.
Are you able to recommend a resource for how to use bandwidth most efficiently like this? I'm interested in learning more, and would appreciate it. Also, might using raw sockets not offer better performance and lower CPU usage?
There is quite a bit of documentation on this particular snake oil: https://patents.justia.com/patent/20090063698 https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol
If Aspera is as good as the parent suggests it's probably because they are operating a bittorrent-like network of geo-distributed peers. It would be very cheap to emulate that using cloud computing providers, especially if you are discarding the data after transfer.
A proprietary protocol that somehow facilities throughout orders of magnitude than non proprietary alternatives. I call BS.
But, I think Aspera uses some specialized compression for genomics data, and genomics compression is not easy.
My suggestion would be to talk to the telco that provides your wan. If you are using a direct internet connection my suggestion would be : don't.