Live data from Hacker News

Ask HN: What is your favorite method of sending large files?

news.ycombinator.com

331–340 of 363 posts

Re: Ask HN: What is your favorite method of sending large files?

#331
post #2

Magic Wormhole ( https://magic-wormhole.readthedocs.io/en/latest/ ) if it's between OSes. Between OSX, AirDrop works very well. I have sent >10GB files between Macs, quite quick as well. I have never send a 10TB file so I wouldn't know. None of my drives are that large yet :)

This is local lan only right? Or are you port forwarding to expose it?

Re: Ask HN: What is your favorite method of sending large files?

#332
post #312

Earlier quoted context omitted.

BitTorrent is the best for sending large files. I wish there existed a tool to make torrent creation easier, and I also wish my computer could seed. Every time I create a torrent, nobody can discover my server, even though the port is open and even if I add the server IP as a peer directly onto the client. I have no trouble seeding in swarms, only when I create torrents. Other than that, BT is ideal.

Maybe you're only able to make outbound connections and not able to accept incoming connections? Try port scanning your PC from the internet

No, connecting externally is fine, that's what's so weird about it.

Re: Ask HN: What is your favorite method of sending large files?

#333
post #295
post #180

Earlier quoted context omitted.

I initiated the croc project. Others have asked me the same thing, so I'll bite: I chose not to copy the features of magic wormhole because it seemed to me (at the time) to be very complicated and lacking in features crucial to myself (namely restarting transfers [1], sending folders without zipping them). This has been to my benefit, I think, because I can iterate on croc quickly without having to conform to the wor…

Can you write what is that hadcoded relay in croc? Who maintains it, how does it handle the increase in demand or outages etc?

I maintain it. Its been up for about 2 years with pretty minimal downtime.

In any case, I encourage everyone to setup their own relay. Its easy. Make sure ports 9009-9014 are open and then run `croc relay`.

Connecting to your relay is as easy as `croc --relay ADDRESS:9009 send X`. And if you add the `--remember` flag you only have to tell it which relay to use once.

Re: Ask HN: What is your favorite method of sending large files?

#334
post #180

Earlier quoted context omitted.

I initiated the croc project. Others have asked me the same thing, so I'll bite: I chose not to copy the features of magic wormhole because it seemed to me (at the time) to be very complicated and lacking in features crucial to myself (namely restarting transfers [1], sending folders without zipping them). This has been to my benefit, I think, because I can iterate on croc quickly without having to conform to the wor…

> so I'll bite Just curious: pun intended?

I didn't see this one until now :)

Re: Ask HN: What is your favorite method of sending large files?

#335
post #180

Earlier quoted context omitted.

I initiated the croc project. Others have asked me the same thing, so I'll bite: I chose not to copy the features of magic wormhole because it seemed to me (at the time) to be very complicated and lacking in features crucial to myself (namely restarting transfers [1], sending folders without zipping them). This has been to my benefit, I think, because I can iterate on croc quickly without having to conform to the wor…

Would it be possible to sniff the protocol on the wire and provide some sort of cross-compatible magic wormcroc service?

Definitely possible. I'll happily take any PR that does this.

Re: Ask HN: What is your favorite method of sending large files?

#336
post #329

Earlier quoted context omitted.

Thank you. Croc is fantastic because it solves a problem that no other tool does. Magic Wormhole stumbles at the first hurdle (installability).

I'm sorry but I don't understand the hurdle? As I mentioned in another thread installing Magic Wormhole is _easy_, just "brew install magic-wormhole" or "sudo apt-get install magic-wormhole" (or distro equivalent). Windows might be tricky but don't see how croc does it much better. I hadn't heard of Scoop but seems like you'd need to install _that_ first (which has its own dependencies). That said I haven't used Wind…

As BusTrainBus says, "installability" is useful, and it was one of my motivations.

I agree installation for magic wormhole is easy for Mac/Linux, but I think its not easy on Windows. Windows is very common (77% of the market share for desktops [1]) so I wanted to keep croc easy to install for Windows.

In fact, if you are just receiving a file on Windows, all you have to do is download the binary from releases [2], unzip it, and double click on the binary - no terminal experience required. (Sending a file does require using a terminal, but all CLI apps have that hurdle...).

[1]: https://gs.statcounter.com/os-market-share/desktop/worldwide

[2]: https://github.com/schollz/croc/releases

Re: Ask HN: What is your favorite method of sending large files?

#337

Netcat: $ nc -l 4242 > dest And then on the sending end: $ nc hostname 4242 This works great when you just need to get a file of any size from one machine to another and you’re both on the same network. Are used this a lot at one of my offices to schlep the files around between a few of the machines we had.

I've done that many times, often in combination with tar and pv as mentioned in another reply. But I prefer to do the listening on the sending side. Feels more like serving than shoving I guess. Also, I remember needing a -q flag on one of the ends for graceful shutdowns.

Re: Ask HN: What is your favorite method of sending large files?

#338
post #178

Earlier quoted context omitted.

wget -c, it ain't hard.

The problem isn't the clients. The HTTP server needs to provide optional support for range requests on that resource. (RFC 7233) Not all do.

Every major webserver supports resume.

Re: Ask HN: What is your favorite method of sending large files?

#339
post #285
post #179

Earlier quoted context omitted.

Just be careful - unless you mark that torrent private it'll get posted to the DHT and crawlers like BtDig will pick it up and list it publicly. For this reason I prefer using something like Syncthing which is designed more with this purpose in mind.

For some reason I haven’t understood, syncthing would often miss transferring files but mark the entire transfer/sync as complete. No errors in the logs either. I tried it for a few months earlier this year, then gave up and switched to plain rsync in a shell. Based on that sample of one personal experience, I wouldn’t recommend syncthing to anyone. Note: I was using syncthing for a one way sync all the time, with se…

I've been using Syncthing extensively for years and never observed this behavior - it always either completes the transfer or fails to complete the transfer (which is obvious in the UI). A silent failure is certainly not typical.

This sounds like a pretty severe bug, so if you still have the systems you were able to produce the behavior on it might be helpful to reproduce it and submit an issue.

Re: Ask HN: What is your favorite method of sending large files?

#340
post #251
post #70

Earlier quoted context omitted.

This is especially fun when combined with piping through tar, and adding pv in the mix for a transfer speed "progress" bar. Fastest way to transfer a collection of files on a local network and doesn't require temporary storage for the archive

What's an example of the syntax of this? I remember seeing a graybeard do something like this to huck some files to another machine.

Paraphrased from a real example:

  tar cf - files morefiles | ssh user@dest "tar xf -"
If you want to insert, say, compression:

  tar cf - files morefiles | zstd | ssh user@dest "zstd -d | tar xf -"
Post reply on HN