Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSH and User-Mode IP WireGuard

fly.io

Post reply on HN
Print view View original

81–90 of 104 posts

Re: SSH and User-Mode IP WireGuard

#81

Post by xmodem »

tptacek wrote: post #79
omribahumi wrote: post #78

Earlier quoted context omitted.

Have you considered using ssh command's ProxyCommand option? It allows you to replace the TCP transport with communication over stdin/stdout. It could help you replace the TUN with something more cross platform, and possibly with less overhead. You can pass in the hostname using %h, so you can even have virtual DNS.

How does that help us here? Without WireGuard, there's no channel with which you can talk to a Fly Hallpass instance, by design.

Implement the userspace wireguard client and TCP stack as a ProxyCommand, make something useful for the general case of SSH'ing over wireguard.

Re: SSH and User-Mode IP WireGuard

#82

Post by smithclay »

Hacking stuff together using a userspace networking stack is an incredibly fun side project and significantly easier with the gVisor networking libraries written in Go.

Last year I implemented TCP/IP over AWS Cloudwatch. Tons of "can you believe that actually works?" stuff possible with it:

https://medium.com/clog/tcp-ip-over-amazon-cloudwatch-logs-c...

Re: SSH and User-Mode IP WireGuard

#83

Post by mwcampbell »

mrkurt wrote: post #48
ignoramous wrote: post #47

Earlier quoted context omitted.

> Regarding the article, it seems like Fly has pulled off some insane networking nonsense Fly is essentially building a Tailscale-esque infrastructure to service one part of their cloud offering. It is indeed insane the amount of heavy-lifting they do to make it all work. They seem like a cross between packetfabric, gitops, docker, and hashicorp but with way less engineers on the team.

To be fair, what Tailscale is doing is much harder than our private networking. They have to deal with NAT, mobile OSes, etc. We mostly just try to pick the right primitives. And frequently get that wrong. Like that time we wrote our own JS runtime ...

> They have to deal with NAT, mobile OSes, etc.

Does this imply that the user-space TCP/IP-over-WireGuard trick described here wouldn't work through NAT, or on a mobile OS (assuming you can get a Go toolchain up and running)?

Re: SSH and User-Mode IP WireGuard

#84

Post by benjaminl »

anderspitman wrote: post #61

This is fantastic. I maintain a list[0] of tunneling software. One of the few downsides of WireGuard is the inability to run it in unprivileged situations. The complexity and performance overhead here might still be too much to edge out solutions like SSH tunnels, but I love that the space is being explored. I'm hopeful we'll also see some robust QUIC-based tunneling tools over the next couple years. [0]: https://git…

With the coming ubiquity of QUIC, its seems natural to have a QUIC based analog to OpenVPN using packet based QUIC instead of OpenVPN’s UDP/TLS.

It also seems rather obvious to extend WireGuard to run over QUIC in addition to UDP. But the movement on that front has been very limited.

Re: SSH and User-Mode IP WireGuard

#85

Post by tucif »

tptacek wrote: post #25
rileymichael wrote: post #24

Pretty cool write up. It mentions that every host is running a DNS server that instances have access to, which is being utilized to store the public key (neat!)... is there any way for customers to consume this for other purposes, say out of the box service (instance) discovery?

Yes; the original purpose of private DNS at Fly was for service discovery. `your-app.internal` is the AAAA's of every instance for your-app; `nrt.your-app.internal` every instance in Japan, `aws-rds-1._peer.internal` is AAAA for the other side of a WireGuard gateway you created to bridge your apps to an RDS database, etc.

When you say "the public key for that root certificate is hosted in our private DNS", does that mean the public key is in.. a txt record?

Re: SSH and User-Mode IP WireGuard

#86

Post by mwcampbell »

tptacek wrote: post #2

I added some example code to the post, because, again, I kind of can't get over how easy this turns out to be. And if you follow the link into Jason's `wireguard-go` code, until you hit gVisor itself, it's not much more complicated under the hood. Having complete control of TCP/IP in userland like this, with so little code, is so valuable I feel like there needs to be some special name for the technique. The whole th…

> Having complete control of TCP/IP in userland like this, with so little code, is so valuable I feel like there needs to be some special name for the technique.

Many years ago, when we could take always-on desktop PCs more or less for granted, I developed a product that let the user connect back to their home PC from another PC, to stream music from home or grab a file (this was also pre-Dropbox). NAT was already ubiquitous by this point, and Windows XP SP2 (first version with Windows Firewall) came out that year, so I knew it couldn't just make a direct TCP connection to the user's home PC. So I did a stupid relay implementation, where both the client and the home server (that's what we actually called the tray applet on the home PC) made outgoing TCP connections to our central server, which would relay packets back and forth. If I'd had access to a TCP-in-userspace thing like the gvisor network stack, I could have run TCP end-to-end, the way it's meant to be used. It almost makes me want to reimplement that old system using Go and WireGuard, even though the functionality is basically irrelevant in today's world.

Re: SSH and User-Mode IP WireGuard

#87

Post by abrookewood »

Man, some people are just next level productive: "How hard could it be to put together a tiny user-mode TCP, just for the purposes of doing pure-userland WireGuard networking, so people could SSH into instances on Fly without installing WireGuard? I made the mistake of musing about this on a Slack channel I share with Jason Donenfeld. I mused about it just before I went to bed. I woke up. Jason had implemented it, using gVisor, and made it part of the WireGuard library."

Re: SSH and User-Mode IP WireGuard

#88

Post by tptacek »

xmodem wrote: post #81
tptacek wrote: post #79

Earlier quoted context omitted.

How does that help us here? Without WireGuard, there's no channel with which you can talk to a Fly Hallpass instance, by design.

Implement the userspace wireguard client and TCP stack as a ProxyCommand, make something useful for the general case of SSH'ing over wireguard.

You can totally do that. The code is public, and for a simple TCP proxy, we're talking, maybe, a couple dozen lines.

Re: SSH and User-Mode IP WireGuard

#89

Post by mrkurt »

mwcampbell wrote: post #83
mrkurt wrote: post #48

Earlier quoted context omitted.

To be fair, what Tailscale is doing is much harder than our private networking. They have to deal with NAT, mobile OSes, etc. We mostly just try to pick the right primitives. And frequently get that wrong. Like that time we wrote our own JS runtime ...

> They have to deal with NAT, mobile OSes, etc. Does this imply that the user-space TCP/IP-over-WireGuard trick described here wouldn't work through NAT, or on a mobile OS (assuming you can get a Go toolchain up and running)?

It'll work through NAT for sure. We just don't have to think about NAT because we're connecting clients to a network we control. Tailscale is making a mesh across clients behind different NATs.

I know they did a bunch of work to get wireguard-go working on iOS. It sounds hard to me!

Re: SSH and User-Mode IP WireGuard

#90

Post by mrkurt »

aleph- wrote: post #75
tptacek wrote: post #65

Earlier quoted context omitted.

I think you've got it. Tailscale is installing WireGuard . You have to have privileges to install Tailscale. They can tell the OS to route packets through their virtual interface. We could too! This is all in `wireguard-go`. But we'd have to prompt users to escalate privileges every time they tried to SSH somewhere (or, worse, install a long-term resident thingy, just to SSH to things ). We don't want to own your VPN…

So I'm curious are there any good documentation available for using wireguard-go as a lib? Or is it just read the source and also read through flyctl source? Curious about fiddling with something similar with firecracker at home. Think it'd be neat to spin up bespoke micro-vm's with wireguard enabled.

The source is about it, it's pretty readable though.

If you're turning up microvms with a linux kernel, it might just be easier to use kernel mode wireguard. It works pretty well!

Post reply on HN