In the enterprise space, there is dedicated software for this function. It uses an UDP protocol designed directly for efficiency (less overhead, vs. TCP), and builds in resilience such that it'll work over connections with lots of packet drops, or connections with high latency or low reliability.
Ask HN: What is your favorite method of sending large files?
311–320 of 363 posts
Re: Ask HN: What is your favorite method of sending large files?
#312Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints. Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice. Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea eith…
BitTorrent is the best for sending large files. I wish there existed a tool to make torrent creation easier, and I also wish my computer could seed. Every time I create a torrent, nobody can discover my server, even though the port is open and even if I add the server IP as a peer directly onto the client. I have no trouble seeding in swarms, only when I create torrents. Other than that, BT is ideal.
Maybe you're only able to make outbound connections and not able to accept incoming connections? Try port scanning your PC from the internet
Re: Ask HN: What is your favorite method of sending large files?
#313Earlier quoted context omitted.
I never understood why croc chose to copy the features of Magic Wormhole but implement them in an incompatible way. Partially out of my annoyance with this I implemented a port of Magic Wormhole in Go that fully interoperates with the official python version: https://github.com/psanford/wormhole-william
I initiated the croc project. Others have asked me the same thing, so I'll bite: I chose not to copy the features of magic wormhole because it seemed to me (at the time) to be very complicated and lacking in features crucial to myself (namely restarting transfers [1], sending folders without zipping them). This has been to my benefit, I think, because I can iterate on croc quickly without having to conform to the wor…
> so I'll bite
Just curious: pun intended?
Re: Ask HN: What is your favorite method of sending large files?
#314Earlier quoted context omitted.
I never understood why croc chose to copy the features of Magic Wormhole but implement them in an incompatible way. Partially out of my annoyance with this I implemented a port of Magic Wormhole in Go that fully interoperates with the official python version: https://github.com/psanford/wormhole-william
I initiated the croc project. Others have asked me the same thing, so I'll bite: I chose not to copy the features of magic wormhole because it seemed to me (at the time) to be very complicated and lacking in features crucial to myself (namely restarting transfers [1], sending folders without zipping them). This has been to my benefit, I think, because I can iterate on croc quickly without having to conform to the wor…
Thank you. Croc is fantastic because it solves a problem that no other tool does. Magic Wormhole stumbles at the first hurdle (installability).
Re: Ask HN: What is your favorite method of sending large files?
#315Large files, and I mean from hundreds of GB to terrabytes I use https://www.filemail.com/ . Their desktop app uses UDP, so it is a lot faster than TCP based protocols like FTP and HTTP. Yeah you can still use their website to send stuff, and if small in size, free anyway for up to 50 GB.
UDP really makes the difference in speed! <3
Re: Ask HN: What is your favorite method of sending large files?
#316Earlier quoted context omitted.
You could also gpg encrypt the file first which is a little more work but bulletproof. Maybe someone should make a tool that creates a torrent and encrypts it for you and presents a password box on the other end.
Like magic warmhole?
Magic Wormhole
Re: Ask HN: What is your favorite method of sending large files?
#317Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints. Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice. Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea eith…
I've written a cross platform gui tool to transfer arbitrary large files via p2p with end-to-end encryption.
- Fully p2p, even the signalling server part which is done via Tor hidden services (later on there will be an alternative using a DHT).
- So unless the Internet or Tor goes offline it can never be down.
- Works with proxies and in corporate environments when UDP might be blocked.
Re: Ask HN: What is your favorite method of sending large files?
#318Earlier quoted context omitted.
TCP indeed has checksums! But they are known not to offer ultimate protection. I had to deal with hosts corrupting TCP streams due to bad hardware before. Which was luckily detected by higher level checksums.
Is there a streamlike tool (a la sed, gzip, etc) that applies forward error correction? I've been looking for such a thing but can't find one that scratches the itch.
[deleted]
Re: Ask HN: What is your favorite method of sending large files?
#319rsync over ssh with retries
Re: Ask HN: What is your favorite method of sending large files?
#320Have a look at the "File Sharing and Synchronization" section
https://github.com/awesome-selfhosted/awesome-selfhosted#fil...