It would be nice if the team could publish a high-level architecture of the system!
I'd like to understand how Nebula works -- any other suggestions besides just diving into the code?
Nebula, Slack's Open Source Global Overlay Network
41–50 of 71 posts
Re: Nebula, Slack's Open Source Global Overlay Network
#42[deleted]
Re: Nebula, Slack's Open Source Global Overlay Network
#43How does this differ from WireGuard?
WireGuard is a VPN, and Nebula is an overlay network (also known as a service mesh). They are closely related concepts. VPNs are primarily used for remote access, to get random machines access to closed IP networks. Service meshes synthesize a new network (sometimes IP, sometimes something else) to connect a bunch of related machines, almost always with policy controls for who can talk to what, usually cryptographic.…
Thanks, that’s a helpful comparison!
Re: Nebula, Slack's Open Source Global Overlay Network
#44I feel like I don't totally follow how you would set this up for, say, a company that has infra in two cloud providers (but no office network or datacenter or anything)... I think the answer is you set up one or more lighthouses with stable IPs on the public internet, and you make sure all your ephemeral cloud machines have IPs on the public internet? And all your ephemeral cloud machines get RFC-1918 addresses that…
(from a single read of the README and the OP) I think the answer is that your lighthouse(s) are the only machines that need publicly routable IPs. Your ephemeral cloud machines get any RFC-1918 address you want, with any subnet you want. Engineers would have Nebula set up on their laptop with a configuration that knows about your lighthouse(s) static IP(s). They use the lighthouses for meeting other nodes, UDP hole p…
NAT traversal sounds like a thing I very much don't want to deal with for a production network, instinctively. It's fine for video games with friends but I've seen enough stuff go wrong with even normal networks that I wouldn't want to trust it. If this is what Slack is actually doing, I'd be very curious to hear how it's working out for them and how they debug network outages.
(Which is why I suspect it's not and the readme isn't clear)
Re: Nebula, Slack's Open Source Global Overlay Network
#45Network virtualization seems to have been extremely slow to be adopted. Even companies pushing "cloud first" seem to be running their physical networks like its 2003.
When your Kubernetes falls over, you can ssh to it and run commands like it's 2003 (or 1993). When your network falls over, not so much. We deprecated our network virtualization at $work and have been way happier for it.
Also, the end-to-end principle argues for putting complicated logic in the endpoints and making the network boring. See also, TCP is implemented at the endpoints and just requires network infrastructure to drop packets sometimes. You could imagine a congestion control protocol implemented on each router on the Internet, but it would be much more fragile and also much harder to deploy changes to.
Re: Nebula, Slack's Open Source Global Overlay Network
#46Can nodes communicate with each other directly even if they're behind NAT, without port mappings or UPnP?
I know there are ways to make this happen (e.g. using the techniques from Samy Kamkar's pwnat/chownat), but am not sure whether Nebula is designed to work within this constraint.
Re: Nebula, Slack's Open Source Global Overlay Network
#47Earlier quoted context omitted.
(from a single read of the README and the OP) I think the answer is that your lighthouse(s) are the only machines that need publicly routable IPs. Your ephemeral cloud machines get any RFC-1918 address you want, with any subnet you want. Engineers would have Nebula set up on their laptop with a configuration that knows about your lighthouse(s) static IP(s). They use the lighthouses for meeting other nodes, UDP hole p…
NAT traversal sounds like a thing I very much don't want to deal with for a production network, instinctively. It's fine for video games with friends but I've seen enough stuff go wrong with even normal networks that I wouldn't want to trust it. If this is what Slack is actually doing, I'd be very curious to hear how it's working out for them and how they debug network outages. (Which is why I suspect it's not and th…
Yeah, after doing a few years of VoIP I pretty much learned the same. Yes, there are multiple methods of traversal, yes they are sound in theory. Yet stuff breaks all the time on consumer routers.
Re: Nebula, Slack's Open Source Global Overlay Network
#48Earlier quoted context omitted.
ZT allows you to run your own "Moons", meaning you dont need their infrastructure... bit more config required on the client end, but less reliance on Zerotier....
We are about to make that much easier and also allow true infrastructure federation -- you won't have to have your nodes contact our roots directly. We came up with something very interesting. As for crypto: we plan some improvements in 2.0, but note that these days easily >90% of the traffic over ZeroTier networks (or any other VPN / overlay) tends to be already encrypted via SSH, SSL, etc. Another layer of encrypti…
OT: Thank you guys for the work you've been doing. Been using ZT for years and looking forward to root server decentralization.
Re: Nebula, Slack's Open Source Global Overlay Network
#49Network virtualization seems to have been extremely slow to be adopted. Even companies pushing "cloud first" seem to be running their physical networks like its 2003.
When your Kubernetes falls over, you can ssh to it and run commands like it's 2003 (or 1993). When your network falls over, not so much. We deprecated our network virtualization at $work and have been way happier for it. Also, the end-to-end principle argues for putting complicated logic in the endpoints and making the network boring. See also, TCP is implemented at the endpoints and just requires network infrastruct…
Part of the reason for the end-to-end argument is to enable more clever (or, at least, more purpose-designed) functionality to ride on top of the dumb network. So e2e would suggest (to my reading at least) that you keep the "real" IP layer dumb and flexible, and do the fun stuff in overlays, which is what this is.
Re: Nebula, Slack's Open Source Global Overlay Network
#50Can nodes communicate with each other directly even if they're behind NAT, without port mappings or UPnP? I know there are ways to make this happen (e.g. using the techniques from Samy Kamkar's pwnat/chownat), but am not sure whether Nebula is designed to work within this constraint.
Having two nodes communicate to each other when you have a cooperating third-party server (a lighthouse or discovery node) that isn't behind a nat isn't hard. That's what STUN servers and other forms of UDP hole punching accomplish.
pwnat is notable because it doesn't require having a public stun-like server, but nebula already assumes there's public servers, so traversing nat is a non-issue.
The readme says "Discovery nodes allow individual peers to find each other and optionally use UDP hole punching to establish connections from behind most firewalls or NATs".
In practice, I didn't see any code that implements it, but I didn't look too hard.