Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magic Wormhole: get things from one computer to another, safely

github.com

Post reply on HN
Print view View original

91–100 of 262 posts

Re: Magic Wormhole: get things from one computer to another, safely

#91

Post by bscphil »

geerlingguy wrote: post #30

I've used this for years when passing large files between systems in weird network environments, it's almost always flawless. For some more exotic testing, I was able to run my own magic wormhole relay[1], which let me tweak some things for faster/more reliable huge file copies. I still hate how often Google Drive will fall over when you throw a 10s-of-GB file at it. [1] https://www.jeffgeerling.com/blog/2023/my-own-…

> For some more exotic testing, I was able to run my own magic wormhole relay[1], which let me tweak some things for faster/more reliable huge file copies.

The lack of improvement in these tools is pretty devastating. There was a flurry of activity around PAKEs like 6 years ago now, but we're still missing:

* reliable hole punching so you don't need a slow relay server

* multiple simultaneous TCP streams (or a carefully designed UDP protocol) to get large amounts of data through long fat pipes quickly

Last time I tried using a Wormhole to transmit a large amount of data, I was limited to 20 MB/sec thanks to the bandwidth-delay product. I ended up using plain old http, with aria2c and multiple streams I maxed out a 1 Gbps line.

IMO there's no reason why PAKE tools shouldn't have completely displaced over-complicated stuff like Globus (proprietary) for long distance transfer of huge data, but here we are stuck in the past.

Re: Magic Wormhole: get things from one computer to another, safely

#93

Post by lotharrr »

AtlasBarfed wrote: post #67
netsec_burn wrote: post #10

I've used wormhole once to move a 70 GB file. Couldn't possibly do that before. And yes, I know I used the bandwidth of the relay server, I donated to Debian immediately afterwards (they run the relay for the version in the apt package).

It took this far down in the comments to get to some inkling of the meat of this. It relys on some singular or small set of donated servers? NAT NAT traversal is obviously the biggest motivator, since otherwise you just scp or rsync or sftp if you don't have the dual barrier. Is the relay server configurable? Seemed to be implied it is somewhat hardcoded.

Yes, it relies on two servers, both of which I run. All connections use the "mailbox server", to exchange short messages, which are used to do the cryptographic negotiation, and then trade instructions like "I want to send you a file, please tell me what IP addresses to try".

Then, to send the bulk data, if the two sides can't establish a direct connection, they fall back to the "transit relay helper" server. You only need that one if both sides are behind NAT.

The client has addresses for both servers baked in, so everything works out-of-the-box, but you can override either one with CLI args or environment variables.

Both sides must use the same mailbox server. But they can use different transit relay helpers, since the helper's address just gets included in the "I want to send you a file" conversation. If I use `--transit-helper tcp:helperA.example.com:1234` and use use `--transit-helper tcp:helperB.example.com:1234`, then we'll both try all of:

* my public IP addresses * your public IP addresses * helperA (after a short delay) * helperB (after a short delay)

and the first one to negotiate successfully will get used.

> since otherwise you just scp or rsync or sftp if you don't have the dual barrier

True, but wormhole also means you don't have to set up pubkey ahead of time.

Re: Magic Wormhole: get things from one computer to another, safely

#94

Post by Twixes »

fn0rd_ wrote: post #78
hoppyhoppy2 wrote: post #3

A similar project with some nice features that I use is croc: https://github.com/schollz/croc

Maybe stay with wormhole https://redrocket.club/posts/croc/

Got fixed pretty thoroughly though, it seems! https://schollz.com/tinker/croc9/

Re: Magic Wormhole: get things from one computer to another, safely

#95

Post by lotharrr »

happosai wrote: post #84
lotharrr wrote: post #48

author here.. happy to answer any questions!

Nice project. Is there iptables connection tracking module that can handle the protocol?

None that I know of. It just uses a TCP connection to the mailbox server (with keepalives), and then TCP connections for the bulk-transfer transit phase, so I can't think of anything special that iptables would need to handle it well.

The encrypted connection is used to exchange IP addresses.. maybe you're thinking of the module that e.g. can modify FTP messages to replace the IP addresses with NAT-translated ones? Our encryption layer would prevent that, but we'd probably get more benefit from implementing WebRTC or a more general hole-punching scheme, than by having the kernel be able to fiddle with the addresses.

Re: Magic Wormhole: get things from one computer to another, safely

#96

Post by bscphil »

lotharrr wrote: post #65
cl3misch wrote: post #64

Earlier quoted context omitted.

> you need a machine that can handle whatever link speeds you need I would have expected the relay server only being used for initial handshake to punch through NAT, after which the transfer is P2P. Only in the case of some network restrictions the data really flows through the relay. How could they afford running the free relay otherwise?

There are two servers. The "mailbox server" helps with handshakes and metadata transfers, and is super-low bandwidth, a few hundred bytes per connection. The "transit relay helper" is the one that handles the bulk data transfer iff the two sides were unable to establish a direct connection. I've been meaning to find the time to add NAT-hole-punching for years, but haven't managed it yet. We'd use the mailbox server m…

The folks on the wormhole-rs fork (who appear to share your Github organization? [1]) already have NAT punching working 95+% of the time in my testing, so maybe what they're doing could be ported over to the Python implementation.

[1] https://github.com/magic-wormhole

Re: Magic Wormhole: get things from one computer to another, safely

#97

Post by from-nibly »

cl3misch wrote: post #64
geerlingguy wrote: post #30

I've used this for years when passing large files between systems in weird network environments, it's almost always flawless. For some more exotic testing, I was able to run my own magic wormhole relay[1], which let me tweak some things for faster/more reliable huge file copies. I still hate how often Google Drive will fall over when you throw a 10s-of-GB file at it. [1] https://www.jeffgeerling.com/blog/2023/my-own-…

> you need a machine that can handle whatever link speeds you need I would have expected the relay server only being used for initial handshake to punch through NAT, after which the transfer is P2P. Only in the case of some network restrictions the data really flows through the relay. How could they afford running the free relay otherwise?

You cant make a p2p connection over a NAT without exposing a port on the public side of the NAT.

Re: Magic Wormhole: get things from one computer to another, safely

#98

Post by lotharrr »

jmakov wrote: post #73

Can it also generate a link so the client can just ckick on it instead of having to install things?

Alas no. It's a one-shot file-transfer tool, and we don't store a copy of the encrypted data or anything. So the sender must stay running until the receiver has finished downloading. If you're comfortable with relying on servers for your security, then I believe wormhole.app offers the clickable zero-install link that you described.

Magic-wormhole can't use that approach, because our security model rules out reliance on servers for confidentiality or integrity. We could safely store ciphertext without violating the model, but you need an interactive protocol with the sender to get the decryption key (otherwise the wormhole code would be a lot larger), so it wouldn't improve the experience very much, and would cost a lot more to operate. The wormhole servers have trivial storage requirements, so the only real costs are bandwidth for the transit relay helper, for when the two sides can't make a direct connection.

Re: Magic Wormhole: get things from one computer to another, safely

#99

Post by j1elo »

I'm reading so many good comments that I will be trying Magic Wormhole soon.

How about Warpinator [1]?

It's the application that I use simply because it came by default with my choice of Linux distro, and it works fine. Main use case for me is sending recently taken photos from my phone to the computer.

[1]: https://github.com/linuxmint/warpinator

Re: Magic Wormhole: get things from one computer to another, safely

#100

Post by t0mas88 »

from-nibly wrote: post #97
cl3misch wrote: post #64

Earlier quoted context omitted.

> you need a machine that can handle whatever link speeds you need I would have expected the relay server only being used for initial handshake to punch through NAT, after which the transfer is P2P. Only in the case of some network restrictions the data really flows through the relay. How could they afford running the free relay otherwise?

You cant make a p2p connection over a NAT without exposing a port on the public side of the NAT.

You can: https://en.wikipedia.org/wiki/Hole_punching_(networking)
Post reply on HN