I have used linux's rsync to send up to 100GiB at a time if the connection is reasonably stable. There are a couple challenges with this method: 1) the way it resumes takes a while to recheck the file before staring again; the larger the file, the longer this takes 2) you need to know a bit about writing a simple script to loop the rsync until the transfer is completed.
Ask HN: What is your favorite method of sending large files?
291–300 of 363 posts
Re: Ask HN: What is your favorite method of sending large files?
#292I upload it to my VPS and send someone a link.
I think google drive is a decent solution as well.
Re: Ask HN: What is your favorite method of sending large files?
#293At 10 TB I guess your best option is to physically mail an HDD.
Re: Ask HN: What is your favorite method of sending large files?
#294If the data is below 50G in size and on my personal computer, then I just drop it into the Google Drive folder and it syncs to Google over night. When it is done, I export it and then send a link to the person(s). I pay $2 a month for the 100G account and I usually have about 50G disk space unused, so this is not an issue for me. If it is above 50G and on my personal computer, I encrypt the data and then physically m…
> If the data is below 50G in size and on my personal computer, then I just drop it into the Google Drive folder
I do the same thing with OneDrive. Of course that's the same thing and I'm just agreeing with your answer, there are any number of DropBox-like services (including DropBox!). OneDrive is nice if you have Office 365 because you can password protect a link and send the password via a separate channel, which makes the process pretty secure (if you trust Microsoft with your data).
Re: Ask HN: What is your favorite method of sending large files?
#295Earlier quoted context omitted.
I never understood why croc chose to copy the features of Magic Wormhole but implement them in an incompatible way. Partially out of my annoyance with this I implemented a port of Magic Wormhole in Go that fully interoperates with the official python version: https://github.com/psanford/wormhole-william
I initiated the croc project. Others have asked me the same thing, so I'll bite: I chose not to copy the features of magic wormhole because it seemed to me (at the time) to be very complicated and lacking in features crucial to myself (namely restarting transfers [1], sending folders without zipping them). This has been to my benefit, I think, because I can iterate on croc quickly without having to conform to the wor…
Can you write what is that hadcoded relay in croc? Who maintains it, how does it handle the increase in demand or outages etc?
Re: Ask HN: What is your favorite method of sending large files?
#296Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints. Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice. Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea eith…
Just be careful - unless you mark that torrent private it'll get posted to the DHT and crawlers like BtDig will pick it up and list it publicly. For this reason I prefer using something like Syncthing which is designed more with this purpose in mind.
That's why you encrypt it or at least make it a password protected archive. That way no one will see the file names in the content list. And without seeders, it's highly unlikely anyone will download it. Torrents work well with 1 seeder, 1 downloader.
Re: Ask HN: What is your favorite method of sending large files?
#297Earlier quoted context omitted.
Sometimes I wonder if the bad connotation due to piracy made this protocol not standard (in a sense, that's not widely adopted, and people need dedicated clients, instead of it working for example in browsers). I don't know enough, maybe it has security issues, or some limits... but the fact is that it's still widely used, by tech literate people, so it still stands the test of time and it must be doing something rig…
In my previous job we used BitTorrent to distribute movies to cinemas. :-)
What an irony! :)
Re: Ask HN: What is your favorite method of sending large files?
#298I have a home nextcloud with a few extra TB of storage. I would put it there and use a share link.
But HTTP is not a great protocol for really large transfers like 10TB. Ideally you'd want something that parallelizes and does hash checks.
Even then, at 10TB you need a 2 gigabit connection to be even competitive with SSD-and-overnight-shipping (about 12hrs).
Re: Ask HN: What is your favorite method of sending large files?
#299Earlier quoted context omitted.
In my (limited) experience, the Greyhound shipping service becomes competitive when shipments are very bulky (like body parts) more so than when they are heavy.
You mean auto body parts, right?
Well, other body parts probably also fit the bulky description. But they'd also be heavy, I guess.
Re: Ask HN: What is your favorite method of sending large files?
#300Earlier quoted context omitted.
You're gonna love magic-wormhole.
It's great, but we're very far from the os-agnostic seemless integrated files sharing. You see, we have protocols like HTTP and the web relies on it, everyone agrees this is how the web works and all OSes have one or several browser able to access it. Something similar works with emails. Yet, we don't have anything to share files between computers, imagine something like bittorrent directly integrated to all OSes, le…
Having a generic "Send to" would only solve a part of it though. There is currently no standard way of connecting the two ends of the same cable to two computers and expect them to exchange files without heavy configuration on both sides. Wireless goes the same way.