Live data from Hacker News

Ask HN: What is your favorite method of sending large files?

news.ycombinator.com

351–360 of 363 posts

Re: Ask HN: What is your favorite method of sending large files?

#351

5GB on my local network: Windows File Share, Rsync, or HTTP depending on source/destination 5GB on Internet: Upload to my OVH VPS and HTTP or Rsync it to its destination 10TB, local or Internet: Physical Hard Drive. Never underestimate the bandwidth of a station wagon full of backup tapes! https://www.tidbitsfortechs.com/2013/09/never-underestimate-... And since you now buy 1TB micro SD cards, so perhaps I'd split th…

Would you share details of the design of your “ohs vps” facility?

Re: Ask HN: What is your favorite method of sending large files?

#353

Earlier quoted context omitted.

It's technically possible, but most browsers don't auto-resume, and most webservers will timeout a request after a certain amount of time has elapsed. You can get around all this with browser extensions, but it's honestly easier to use a more robust protocol (this is in the context of, say, a 10TB file that'll take you a few days to download).

I don't see how anything would be much more robust than HTTP. Any network based protocol can fail at any time, so you're going to have to have some concept of tracking your progress and retrying failed requests. Both are easy to do with HTTP. Can you give a concrete example of something that would work better than HTTP?

Here's an exercise for you:

In your browser, start downloading a test file, like [1]. Turn off your network card for a few seconds, then turn it back on. See what your browser does.

Next, start downloading a large file over Bittorrent, like [2]. Turn off your network card for a few seconds, then turn it back on. See what the client does.

[1] http://speedtest.tele2.net/100GB.zip

[2] https://cdimage.debian.org/debian-cd/current/amd64/bt-dvd/de...

Re: Ask HN: What is your favorite method of sending large files?

#354

Earlier quoted context omitted.

You can also pipe through gpg if you want encryption along the way!

And through gzip/zstd/etc if you want compression!

Note that gpg compresses by default. Which is thy it's so slow. You may wish to use lzop or zstd but also disable gpg's compression.

I usually use ssh (e.g. ssh -L 3333:localhost:3333) and avoid gpg, though I may use lzop depending on link and CPU speed.

Re: Ask HN: What is your favorite method of sending large files?

#355
post #64
post #45

If the data is below 50G in size and on my personal computer, then I just drop it into the Google Drive folder and it syncs to Google over night. When it is done, I export it and then send a link to the person(s). I pay $2 a month for the 100G account and I usually have about 50G disk space unused, so this is not an issue for me. If it is above 50G and on my personal computer, I encrypt the data and then physically m…

>If it is on a server (For example, a side project instance in AWS), then I drop it into S3 bucket, export it and send the URL to the recipient. I just eat the cost of them downloading it. Usually I am making money off the work anyways, so it is the cost of doing business. Consider BunnyCDN way cheaper than s3.

If the project is already in AWS then it has to be downloaded outside the network at least once. I think s3 is one of the cheapest methods to server files on aws. There is no benefit to a secondary service.

BunnyCDN does look good though.

Re: Ask HN: What is your favorite method of sending large files?

#356
post #222
post #186

Earlier quoted context omitted.

I don't see an option to send text (browsed through README and Issues), is this not possible with croc? With Magic Wormhole it's the default to get a message prompt when running "wormhole send". Or you can do it as a oneliner with "wormhole send --text 'something'". Wormhole William can do this as well IIRC. I use MWH often for sending stuff like URL's, passwords, API keys... having to create a file for that is quite…

Just added it now. Send with croc send --text "hello, world" [1] [1]: https://github.com/schollz/croc/releases/tag/v8.2.0

Nice! I just tried it out now. I also like the fact that "croc" defaults to receive, where with MWH you need to type "wormhole receive". Saves a lot of keystrokes.

Tab completion is something I'll miss from MWH (I'm already converted!) but I can live with it, and my non-technical friends aren't even aware of the concept.

Re: Ask HN: What is your favorite method of sending large files?

#357
post #19

Earlier quoted context omitted.

IMO, GPG is simply not a good tool and it should be replaced. Others have said it better than I can. See e.g. https://latacora.micro.blog/2019/07/16/the-pgp-problem.html

That's said often about any tool which has been around long enough. People without experience come around and think they can replace an old tool with a better one, but it's usually only ignorance of either the complexity of the task, knowledge of using the tool properly, or both.

Disagree.

There are many tools which are way older than gpg, but still in the wide use. For example, I have not heard any arguments that people should stop using things like "rsync" or "curl".

Re: Ask HN: What is your favorite method of sending large files?

#358
post #349

Earlier quoted context omitted.

Sometimes I wonder if the bad connotation due to piracy made this protocol not standard (in a sense, that's not widely adopted, and people need dedicated clients, instead of it working for example in browsers). I don't know enough, maybe it has security issues, or some limits... but the fact is that it's still widely used, by tech literate people, so it still stands the test of time and it must be doing something rig…

> the bad connotation due to piracy made this protocol not standard Back when Resilio Sync was known as Bittorrent Sync, I proposed it as a file-sharing solution between multiple locations for the company I was working for. Immediately returned with an emphatic refusal to use any kind of software that had "Bittorrent" in the name, regardless of what it was for. This is the same company where they picked up I had "Bit…

I was kicked out of the public library once for running Syncthing, I guess because the firewall detected it as some sort of piracy tool

Re: Ask HN: What is your favorite method of sending large files?

#359
post #245

Magic wormhole - it traverses NATs, is encrypted, requires configuration of the source or destination. https://github.com/warner/magic-wormhole

Do you mean requires no configuration on the source or destination? :)

:p little configuration

Re: Ask HN: What is your favorite method of sending large files?

#360
post #245

Earlier quoted context omitted.

Do you mean requires no configuration on the source or destination? :)

Installing it on windows required me to install visual c++ build tools, which is a beast of 1.2GB. So yeah it requires a lot. I rather recommend croc, just dropped it on my path it Just works.

Being on windows is its own reward.

Snark aside, Croc does look a lot like magic wormhole. I'll definitely have a closer look at it.

Post reply on HN