Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One way to make containers network: BGP

jvns.ca

Post reply on HN
Print view View original

21–30 of 71 posts

Re: One way to make containers network: BGP

#21

Post by kijiki »

tptacek wrote: post #20

Especially since there isn't really a policy-routing component to this, isn't BGP pretty _extremely_ complicated for the problem Calico is trying to solve? Stipulating that you need a routing protocol here (you don't, right? You can do proxy ARP, or some more modern equivalent of proxy ARP.), there's a whole family of routing protocols optimized for this scenario, of which OSPF is the best-known.

One reason you sometimes "need" BGP for this is because the networking team is highly skeptical about their ToR switches accepting routes from the server team's hypervisors/containervisors. BGP route filtering on the ToRs makes them feel more secure and happy.

Opinions vary whether this is a real concern, or just a way for the networking team to maintain their relevance.

Re: One way to make containers network: BGP

#22

Post by delinka »

Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router.

BGP is the process by which ranges of IPs are claimed by routers. Is Calico really used by docker containers in this way?

Re: One way to make containers network: BGP

#23

Post by moondev »

amouat wrote: post #19
moondev wrote: post #8

Earlier quoted context omitted.

Is flannel used in Kubernetes for networking by default? Or is it something that needs to enabled and configured separately?

Kubernetes has a requirement that containers (more accurately "pods") can connect via a "flat networking space". How this is achieved varies between deployments, flannel, calico and weave are all common approaches. Kelsey Hightower's "Kubernetes the Hard Way" simply configured it at the router level: https://github.com/kelseyhightower/kubernetes-the-hard-way/b...

That makes sense. Thanks for the link as well, i've been looking for something exactly like it. Looks like a great resource!

Re: One way to make containers network: BGP

#24

Post by lobster_johnson »

moondev wrote: post #8
chrissnell wrote: post #6

It's really not that difficult to network containers. We're using flannel [1] on CoreOS. We're using flannel's VXLAN backend to encapsulate container traffic. We're Kubernetes users so every kube pod [2] gets it's own subnet and flannel handles the routing between those subnets, across all CoreOS servers in the cluster. I was skeptical when we first deployed it but we've found it to be dependable and fast. We're runn…

Is flannel used in Kubernetes for networking by default? Or is it something that needs to enabled and configured separately?

Kubernetes doesn't have a "default" as such. It requires something external to manage the subnet, and needs to be configured to use it.

However, if you run it on AWS, it can automatically configure a bridge (cbr0) and configure up the VPC routing table for you.

GCE (Google's managed Kubernetes on Google Cloud) also handles this automatically.

There's also experimental support for Flannel built into K8s, which can be enabled with a flag. Not sure if it's worth using.

Re: One way to make containers network: BGP

#25

Post by lobster_johnson »

BGP looks really complex. Isn't OSPF (BGP's "little brother") a much attractive choice here? It's still complex, but should be much simpler.

Another attractive alternative to Flannel is Weave [1], run in the simpler non-overlay mode. In this mode, it won't start a SDN, but will simply act as a bridge/route maintainer, similar to Flannel.

[1] https://www.weave.works/products/weave-net/

Re: One way to make containers network: BGP

#26

Post by bboreham »

lobster_johnson wrote: post #24
moondev wrote: post #8

Earlier quoted context omitted.

Is flannel used in Kubernetes for networking by default? Or is it something that needs to enabled and configured separately?

Kubernetes doesn't have a "default" as such. It requires something external to manage the subnet, and needs to be configured to use it. However, if you run it on AWS, it can automatically configure a bridge (cbr0) and configure up the VPC routing table for you. GCE (Google's managed Kubernetes on Google Cloud) also handles this automatically. There's also experimental support for Flannel built into K8s, which can be…

Nitpick: Google's managed Kubernetes is called GKE.

However the OSS Kubernetes has code to configure routes on GCE same as it does for AWS.

Re: One way to make containers network: BGP

#28

Post by bboreham »

crb wrote: post #9

On the topic of "why do we need a distributed KV store for an overlay network?" from the blog: there's a good blog post about why Kubernetes doesn't use Docker's libnetwork. http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-...

Author seems to have missed that Calico also depends on etcd.

Re: One way to make containers network: BGP

#30

Post by bboreham »

jaytaylor wrote: post #14
crb wrote: post #9

On the topic of "why do we need a distributed KV store for an overlay network?" from the blog: there's a good blog post about why Kubernetes doesn't use Docker's libnetwork. http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-...

Thanks! I had almost forgotten about the container networking mayhem. Would love to find out what progress has been made over the past six months since the blog post was written.

Kubernetes now uses CNI to configure interfaces in its most common "kubenet" configuration, and (obviously) when you put it into "CNI" mode.

Most container network offerings - Calico, Flannel, Weave, etc. ship with a CNI plugin.

Docker have not altered their network plugin API.

(I work on Weave Net, including the plugins for both Docker and CNI)

Post reply on HN