Live data from Hacker News

Container Networking with Vxlan, BGP and WireGuard

flockport.com

31–40 of 45 posts

Re: Container Networking with Vxlan, BGP and WireGuard

#31

Earlier quoted context omitted.

> just talk to each other using wiregaurd by default I don't understand why you'd want to do this? I use wireguard to join machines on disparate networks into one. However to do it inside the same VPC, I just don't get. If you don't trust your VPC surely you need to be moving off the cloud?

I agree with your viewpoint but I'm also aware of several security standards that explicitly specify all traffic between hosts needs to be encrypted. Sometimes it's easier to meet the standard verbatim than try and justify an exception. If you already use a configuration management tool it shouldn't be a lot more overhead to install some certificates.

If you think about these things like physical networks, you can do things like run an interface in promiscious mode and sniff traffic.

Further, leaving your VM, you hit a shared NIC and network cables, so you start to worry about phyiscal layer attacks.

Amazon specifically states they handle these issues, and indeed they likely do, but how do you know? If you're able to easily encrypt by using something like istio, then why not?

More specifically:

"Packet sniffing by other tenants: It is not possible for a virtual instance running in promiscuous mode to receive or“sniff” traffic that is intended for a different virtual instance. While customers can place their interfaces into promiscuous mode, the hypervisor will not deliver any traffic to them that is not addressed to them. This includes two virtual instances that are owned by the same customer, even if they are located on the same physical host. Attacks such as ARP cache poisoning do not work within EC2. While Amazon EC2 does provide ample protection against one customer inadvertently or maliciously attempting to view another’s data, as a standard practice customers should encrypt sensitive traffic."

Re: Container Networking with Vxlan, BGP and WireGuard

#32
post #23
post #6

This article uses Quagga - they really should be using FRRouting, which was forked from Quagga in 2017 by the core Quagga developers and has 4 times as many commits (16000[0] vs 4000[1]), far more features, bugfixes, etc. Quagga has been dead for over a year. [0] https://github.com/FRRouting/frr [1] http://gogs.quagga.net/Quagga

For a more full fledged use case FRRouting may be the way to go. We are using Quagga here mainly to maintain internal routes with iBGP and route reflectors. This is a fairly simple use case. Quagga is available in the default package managers of most distros so its a good place to start.

FYI, FRR is in Debian 10, Ubuntu 18.04+, Alpine, soon OpenBSD and Fedora :-)

And they have their own apt repo at deb.frrouting.org for other Debians.

Re: Container Networking with Vxlan, BGP and WireGuard

#33
post #29
post #4

Earlier quoted context omitted.

IPv6 doesn't save you from any routing problems that IPv4 won't save you from. While IPv6 tries to hide the layer 2/layer 3 distinction from you, it doesn't actually make your physical network magically work differently. Internally IPv6 tries to implement this hiding using multicast - same as the VXLAN suggestion in the article. If you overload your network infrastructure's multicast support, at best you fall back to…

> If you don't have multicast support at all in your network infrastructure, which as the article points out isn't common to have on cloud networks, Huh? Are you assuming large flat L2 networks addressed with IPv6? IPv6 works great at scale, just route everything everywhere, stick with unicast & anycast, and don't roll large L2 domains. Multicast is entirely unnecessary aside from the small amount needed for ND/RA be…

Yes, I am. The sort of network architecture you describe works great, but unless I'm seriously misunderstanding the article (which maybe I am!), all the use cases where you'd want VXLAN or Wireguard on IPv4 are incompatible with such an architecture on IPv6.

I will grant that IPv6 + ULAs + BGP + flat networks is easier to think about than IPv4 + 10.0.i++.0/24 + BGP + flat network because you have basically unlimited ULAs, but "You have to pick a unique 10.0.i++.0 for each machine, and that's annoying" doesn't seem like the primary thing the article is trying to forget. If you can do a hierarchical routed IPv6 network, you can almost certainly do it with IPv4, too.

Re: Container Networking with Vxlan, BGP and WireGuard

#34

One does not simply go from a flat network to overlays. Overlays are slow, difficult, cause really odd failures and are often hilariously immature. They are the experimental graph database of the network world. Just have a segregated network, and let the VPC/dhcp do all the hard stuff. Have your hosts on the default VLAN(or Interface if your cloudy), with its own subnet (Subnets should only exist in one VLAN.) Then i…

I will have to take the other side of that golden rule. Not sure where it came from. But when one has a decent handle on the tools at hand, they work wonderously well.

I have bare metal servers tied together with L3 routing via Free Range Routing running BGP/VxLAN. It Just Works.

No hard coded vlans between physical machines. Just point-point L3 links. Vlans are tortuous between machines as a Layer 2 protocol, given spanning tree and all of its slow to converge madness.

Therefore a different Golden Rule:

Always. Overlay. Your. Network.

Leave a note if you'd like more details.

Re: Container Networking with Vxlan, BGP and WireGuard

#35
post #18

In my mind, a "layer 2 subnet" really doesn't mean anything. Subnets are things that happen in IP, that is, layer 3, and layer 2 is the physical connection, ie. Ethernet or WLAN, which don't have the concept of subnets. Edit: also the OSI layer model was specified in the eighties, and isn't all that accurate in 2019 to describe how our networks actually work.

I'd argue that the closest thing to a layer 2 subnet is a VLAN.

But there isn't a one to one relationship.

A subnet should only be in one vlan, but there are networks where there is more than one subnet in a vlan.

Whether that is appropriate or not, that would be a different topic.

Re: Container Networking with Vxlan, BGP and WireGuard

#36

One does not simply go from a flat network to overlays. Overlays are slow, difficult, cause really odd failures and are often hilariously immature. They are the experimental graph database of the network world. Just have a segregated network, and let the VPC/dhcp do all the hard stuff. Have your hosts on the default VLAN(or Interface if your cloudy), with its own subnet (Subnets should only exist in one VLAN.) Then i…

> is no need to faff with routing, it comes for free with your VPC/network or similar

> Always. Avoid. Network. Overlays.

What do you think VPC is?

Re: Container Networking with Vxlan, BGP and WireGuard

#37
"Vxlan uses multicast which is often not supported on most cloud networks. So its best used on your own networks."

Not entirely correct.

Linux has had unicast vxlan for quite some time.

Flannel is doing unicast and works pretty much anywhere.

See "Unicast with dynamic L3 entries" section: https://vincent.bernat.ch/en/blog/2017-vxlan-linux

Re: Container Networking with Vxlan, BGP and WireGuard

#38

"Vxlan uses multicast which is often not supported on most cloud networks. So its best used on your own networks." Not entirely correct. Linux has had unicast vxlan for quite some time. Flannel is doing unicast and works pretty much anywhere. See "Unicast with dynamic L3 entries" section: https://vincent.bernat.ch/en/blog/2017-vxlan-linux

VXLAN is just encapsulating L2 VLANs in UDP packets. Sounds like some confusion about linux implementation details.

Re: Container Networking with Vxlan, BGP and WireGuard

#39

One does not simply go from a flat network to overlays. Overlays are slow, difficult, cause really odd failures and are often hilariously immature. They are the experimental graph database of the network world. Just have a segregated network, and let the VPC/dhcp do all the hard stuff. Have your hosts on the default VLAN(or Interface if your cloudy), with its own subnet (Subnets should only exist in one VLAN.) Then i…

Where I work we use overlays (flannel) and it just works. I don't think we've had issues. AFAIK the primary reason was that the network can be secure/encrypted. Otherwise you're running everything with TLS and managing all the certs can be more painful. Or you're running without encryption which is a potential security problem. You still need to do that for external facing stuff but that's a lot less.

Re: Container Networking with Vxlan, BGP and WireGuard

#40

One does not simply go from a flat network to overlays. Overlays are slow, difficult, cause really odd failures and are often hilariously immature. They are the experimental graph database of the network world. Just have a segregated network, and let the VPC/dhcp do all the hard stuff. Have your hosts on the default VLAN(or Interface if your cloudy), with its own subnet (Subnets should only exist in one VLAN.) Then i…

> is no need to faff with routing, it comes for free with your VPC/network or similar > Always. Avoid. Network. Overlays. What do you think VPC is?

A network overlay written by people that know what they are doing.

but, unless you have an actual reason, why put another layer over the top? especially given the performance and tooling hit.

Post reply on HN