5GB on my local network: Windows File Share, Rsync, or HTTP depending on source/destination 5GB on Internet: Upload to my OVH VPS and HTTP or Rsync it to its destination 10TB, local or Internet: Physical Hard Drive. Never underestimate the bandwidth of a station wagon full of backup tapes! https://www.tidbitsfortechs.com/2013/09/never-underestimate-... And since you now buy 1TB micro SD cards, so perhaps I'd split th…
Ask HN: What is your favorite method of sending large files?
351–360 of 363 posts
Re: Ask HN: What is your favorite method of sending large files?
#352Re: Ask HN: What is your favorite method of sending large files?
#353Earlier quoted context omitted.
It's technically possible, but most browsers don't auto-resume, and most webservers will timeout a request after a certain amount of time has elapsed. You can get around all this with browser extensions, but it's honestly easier to use a more robust protocol (this is in the context of, say, a 10TB file that'll take you a few days to download).
I don't see how anything would be much more robust than HTTP. Any network based protocol can fail at any time, so you're going to have to have some concept of tracking your progress and retrying failed requests. Both are easy to do with HTTP. Can you give a concrete example of something that would work better than HTTP?
In your browser, start downloading a test file, like [1]. Turn off your network card for a few seconds, then turn it back on. See what your browser does.
Next, start downloading a large file over Bittorrent, like [2]. Turn off your network card for a few seconds, then turn it back on. See what the client does.
[1] http://speedtest.tele2.net/100GB.zip
[2] https://cdimage.debian.org/debian-cd/current/amd64/bt-dvd/de...
Re: Ask HN: What is your favorite method of sending large files?
#354Earlier quoted context omitted.
You can also pipe through gpg if you want encryption along the way!
And through gzip/zstd/etc if you want compression!
I usually use ssh (e.g. ssh -L 3333:localhost:3333) and avoid gpg, though I may use lzop depending on link and CPU speed.
Re: Ask HN: What is your favorite method of sending large files?
#355If the data is below 50G in size and on my personal computer, then I just drop it into the Google Drive folder and it syncs to Google over night. When it is done, I export it and then send a link to the person(s). I pay $2 a month for the 100G account and I usually have about 50G disk space unused, so this is not an issue for me. If it is above 50G and on my personal computer, I encrypt the data and then physically m…
>If it is on a server (For example, a side project instance in AWS), then I drop it into S3 bucket, export it and send the URL to the recipient. I just eat the cost of them downloading it. Usually I am making money off the work anyways, so it is the cost of doing business. Consider BunnyCDN way cheaper than s3.
BunnyCDN does look good though.
Re: Ask HN: What is your favorite method of sending large files?
#356Earlier quoted context omitted.
I don't see an option to send text (browsed through README and Issues), is this not possible with croc? With Magic Wormhole it's the default to get a message prompt when running "wormhole send". Or you can do it as a oneliner with "wormhole send --text 'something'". Wormhole William can do this as well IIRC. I use MWH often for sending stuff like URL's, passwords, API keys... having to create a file for that is quite…
Just added it now. Send with croc send --text "hello, world" [1] [1]: https://github.com/schollz/croc/releases/tag/v8.2.0
Tab completion is something I'll miss from MWH (I'm already converted!) but I can live with it, and my non-technical friends aren't even aware of the concept.
Re: Ask HN: What is your favorite method of sending large files?
#357Earlier quoted context omitted.
IMO, GPG is simply not a good tool and it should be replaced. Others have said it better than I can. See e.g. https://latacora.micro.blog/2019/07/16/the-pgp-problem.html
That's said often about any tool which has been around long enough. People without experience come around and think they can replace an old tool with a better one, but it's usually only ignorance of either the complexity of the task, knowledge of using the tool properly, or both.
There are many tools which are way older than gpg, but still in the wide use. For example, I have not heard any arguments that people should stop using things like "rsync" or "curl".
Re: Ask HN: What is your favorite method of sending large files?
#358Earlier quoted context omitted.
Sometimes I wonder if the bad connotation due to piracy made this protocol not standard (in a sense, that's not widely adopted, and people need dedicated clients, instead of it working for example in browsers). I don't know enough, maybe it has security issues, or some limits... but the fact is that it's still widely used, by tech literate people, so it still stands the test of time and it must be doing something rig…
> the bad connotation due to piracy made this protocol not standard Back when Resilio Sync was known as Bittorrent Sync, I proposed it as a file-sharing solution between multiple locations for the company I was working for. Immediately returned with an emphatic refusal to use any kind of software that had "Bittorrent" in the name, regardless of what it was for. This is the same company where they picked up I had "Bit…
Re: Ask HN: What is your favorite method of sending large files?
#359Re: Ask HN: What is your favorite method of sending large files?
#360Earlier quoted context omitted.
Do you mean requires no configuration on the source or destination? :)
Installing it on windows required me to install visual c++ build tools, which is a beast of 1.2GB. So yeah it requires a lot. I rather recommend croc, just dropped it on my path it Just works.
Snark aside, Croc does look a lot like magic wormhole. I'll definitely have a closer look at it.