Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One way to make containers network: BGP

jvns.ca

Post reply on HN
Print view View original

41–50 of 71 posts

Re: One way to make containers network: BGP

#41

Post by X-Istence »

tptacek wrote: post #20

Especially since there isn't really a policy-routing component to this, isn't BGP pretty _extremely_ complicated for the problem Calico is trying to solve? Stipulating that you need a routing protocol here (you don't, right? You can do proxy ARP, or some more modern equivalent of proxy ARP.), there's a whole family of routing protocols optimized for this scenario, of which OSPF is the best-known.

Calico is installing routes in the Linux kernel. Those routes are pulled out and distributed using BIRD. BIRD can do OSPF instead if you'd like.

All Calico cares about is that routes are distributed across various systems, they don't necessarily care how you do it (configure BIRD however you'd like).

BGP is surprisingly simple and easy to set up with BIRD. Setting up a route reflector with local hosts on the same L2 all peering with each other and suddenly you can route whatever IP's you want by announcing them to your peers.

Why do people think BGP is complicated?

Re: One way to make containers network: BGP

#42

Post by X-Istence »

jlgaddis wrote: post #13

BGP seems a needlessly complex solution to this problem. VXLAN would, IMO, be a much better fit. (--Network engineer who manages BGP for an ISP)

Why is BGP complex? What makes BPG complex? The only complex thing about BGP is the fact that you can have it talk to the rest of the world if necessary and deal with traffic balancing and all that fun stuff, dealing with getting routes from multiple locations and a variety of other tasks and topics that are not "BGP" so much as network engineering.

That's not required here.

If you give me 10 machines on a L2 domain, I can set up a private network on top of those 10 machines, and advertise what IP is where to each of them by sharing a routing table... I can of course manually add routes saying a /24 is located on Host 1, and another /24 is on Host 2, or...

What better way to share a routing table than with a route distribution protocol of some sort?

So plop BIRD with BGP on all of the machines, peer em, and have them pull routes out of the Linux routing table and insert routes as necessary.

Now if I spin up a container on Host 1, I advertise a /32 for that IP, and Host 2-10 can all know to forward packets for that /32 to Host 1. If I move that container or IP to Host 2, BGP announces it to all the other hosts and traffic starts flowing there instead.

There is no requirement that Calico (or BIRD rather) peer with any existing BGP infrastructure... you can of course do that, but there is no requirement.

Stop making BGP sound like it's some bad evil thing that's difficult to understand.

Re: One way to make containers network: BGP

#43

Post by q3k »

NetStrikeForce wrote: post #39

Or you could NAT on the host and deploy simpler overlay networking: https://github.com/pjperez/docker-wormhole You can deploy this on any machine (container or not) and have it always reachable from other members of the same network, which could be e.g. servers on different providers (AWS, Azure, Digital Ocean, etc)

You should probably mention that this is a PaaS.

(and maybe also that you are affiliated with them)

Re: One way to make containers network: BGP

#44

Post by jauer »

iheartmemcache wrote: post #35
delinka wrote: post #22

Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router. BGP is the process by which ranges of IPs are claimed by routers. Is Calico reall…

Yeah, have an upvote. This is totally a case of "using an industrial core drill with tungsten carbide bits when all you need is an Ikea drill and a chinese cheese-grade bit". Sounds like this guy just found out about a cool new tool and decided to blog about it. BGP can be used on local networks but it's total overkill for a docker situation where all of your instances are likely in the same rack (often on the same m…

(I didn't downvote you, but this might be why someone did...)

Eh, BGP on local networks is common enough that it was novel maybe a decade ago. It's perfect for running on servers to announce /32 addresses upstream to ToR switches. OSPF is actually more heavyweight & complex since you have to carry link state, do neighbor elections, etc. Ref: NANOG talk in 2003: https://www.nanog.org/meetings/nanog29/presentations/miller....

You don't even need an AS from your RIR for BGP to be useful on internal networks, just pick one (or more) of the private ASNs and roll with it.

Current best practice for internal networks (on the SP side at least) is to use OSPF to bootstrap BGP by enabling IP reachability for p2p & loopback addresses. After that customer / service routes are carried by BGP, not by OSPF. This is because BGP doesn't spam link state and has better policy knobs. You get a double win because OSPF converges faster with fewer routes and if you have redundant links your BGP sessions won't flap because the loopback addresses stay reachable. Ref: NANOG talk in 2011: https://www.nanog.org/meetings/nanog53/presentations/Sunday/...

CPU/RAM overhead is insignificant with a bgpd like BIRD or Quagga. They work for internet-scale routing (currently over 610,000 prefixes) with trivial load. An Atom C-series with a few GB of RAM can deal with internet BGP routing updates (CPU becomes significant for packet forwarding, not so much maintaining routing tables).

I'll take a boring 10-year-old routing setup using iBGP on cluster of route reflectors running BIRD with servers numbered out of a /24 per rack routed by a ToR l3switch with each service using BGP to announce a /32 over new and exciting L2 overlay networks any day. Troubleshooting is easier without having to work through different layers of encapsulation, dealing with MTU, trusting a novel control plane for whatever overlay network your using, etc.

Re: One way to make containers network: BGP

#45

Post by q3k »

delinka wrote: post #22

Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router. BGP is the process by which ranges of IPs are claimed by routers. Is Calico reall…

This works as long as you're willing to stretch an L2 network to reach all the hosts of your container platform. That doesn't really scale.

Using a real routing protocol also immediately gives you access to traffic control, shaping, monitoring and redundancy tools, hardware support and knowledge that network administrators have been applying for years.

Re: One way to make containers network: BGP

#46

Post by cthalupa »

amouat wrote: post #18
cthalupa wrote: post #15

There's a lot of misinformation in this. >A Linux container is a process, usually with its own filesystem attached to it so that its dependencies are isolated from your normal operating system. In the Docker universe we sometimes talk like it's a virtual machine, but fundamentally, it's just a process. Like any process, it can listen on a port (like 30000) and do networking. A container isn't a process. It's an amalg…

I've never read any of her blogs before (that I can recall) and I do agree there are some misunderstandings. But it was clearly written as as a brain dump to help other people going through the same process and it largely achieves that goal. I really don't like the concept "don't write a blog unless you're an expert" - we would lose out on lots of valuable discussions and helpful articles, especially for beginners. F…

I'm certainly not suggesting the need to be an expert to write a blog - I just personally would be more cautious with some statements if I was writing on something I wasn't knowledgable in.

Re: One way to make containers network: BGP

#47

Post by sargun »

bboreham wrote: post #33
sargun wrote: post #16

More on this here: https://medium.com/@sargun/a-critique-of-network-design-ff85... -- BGP isn't just about containers. It's about signaling. It's a mechanism for machines to influence the flow of traffic in the network. This isn't container weirdness. This is because networks got stuck in 2008. We still don't have have IPv6 SLAAC. Many of us made the jump to layer 3 clos fabrics, but stopped after that. My belief is…

> what's wrong with the Docker approach of rewriting ports It requires that you rewrite the software trying to talk to that port, to make it aware that you've put the new port number in a special environment variable.

Have you looked at Docker bridge mode? and Mesosphere's VIPs?

What do you think of them?

Re: One way to make containers network: BGP

#48

Post by gonzo »

jauer wrote: post #44
iheartmemcache wrote: post #35

Earlier quoted context omitted.

Yeah, have an upvote. This is totally a case of "using an industrial core drill with tungsten carbide bits when all you need is an Ikea drill and a chinese cheese-grade bit". Sounds like this guy just found out about a cool new tool and decided to blog about it. BGP can be used on local networks but it's total overkill for a docker situation where all of your instances are likely in the same rack (often on the same m…

(I didn't downvote you, but this might be why someone did...) Eh, BGP on local networks is common enough that it was novel maybe a decade ago. It's perfect for running on servers to announce /32 addresses upstream to ToR switches. OSPF is actually more heavyweight & complex since you have to carry link state, do neighbor elections, etc. Ref: NANOG talk in 2003: https://www.nanog.org/meetings/nanog29/presentations/mil…

> CPU/RAM overhead is insignificant with a bgpd like BIRD or Quagga. They work for internet-scale routing (currently over 610,000 prefixes) with trivial load. An Atom C-series with a few GB of RAM can deal with internet BGP routing updates (CPU becomes significant for packet forwarding, not so much maintaining routing tables).

We've got a DPDK-enabled VRouter that will run over 12Mpps on a C2758 (8 core) with a full BGP table.

Re: One way to make containers network: BGP

#49

Post by chris_marino »

Another solution to this problem is Romana [1] (I am part of this effort). It avoids overlays as well as BGP because it aggregate routes. It uses its own IP address management (IPAM) to maintain the route hierarchy.

The nice thing about this is that nothing has to happen for a new pod to be reachable. No /32 route distribution or BGP (or etcd) convergence, no VXLAN ID (VNID) distribution for the overlay. At some scale, route and/or VNID distribution is going to limit the speed at which new pods can be launched.

One other thing not mentioned in the blog post or in any of these comments is network policy and isolation. Kubernetes v1.3 includes the new network APIs that let you isolate namespaces. This can only be achieved with a back end network solution like Romana or Calico (some others as well).

[1] romana.io

Re: One way to make containers network: BGP

#50

Post by tptacek »

X-Istence wrote: post #41
tptacek wrote: post #20

Especially since there isn't really a policy-routing component to this, isn't BGP pretty _extremely_ complicated for the problem Calico is trying to solve? Stipulating that you need a routing protocol here (you don't, right? You can do proxy ARP, or some more modern equivalent of proxy ARP.), there's a whole family of routing protocols optimized for this scenario, of which OSPF is the best-known.

Calico is installing routes in the Linux kernel. Those routes are pulled out and distributed using BIRD. BIRD can do OSPF instead if you'd like. All Calico cares about is that routes are distributed across various systems, they don't necessarily care how you do it (configure BIRD however you'd like). BGP is surprisingly simple and easy to set up with BIRD. Setting up a route reflector with local hosts on the same L2…

Because BGP is complicated compared to intradomain routing protocols.
Post reply on HN