Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One way to make containers network: BGP

jvns.ca

Post reply on HN
Print view View original

11–20 of 71 posts

Re: One way to make containers network: BGP

#11

Post by api »

jeff_marshall wrote: post #10
api wrote: post #5

Earlier quoted context omitted.

Nothing against that either but some people can't do it due to hybrid deployments, providers that don't give you a /64, or providers that don't offer V6 at all. Currently Amazon, Google, and Azure have no native IPv6 support. Also many are allergic to the security implications. You have to be rigorous with ip6tables and making sure everything speaks SSL or another encrypted protocol using authentication in both direc…

Which provider won't give you a /64 (that supports IPv6 at all)? Even my residential cable provider gives me a /60. Anything less seems absurd.

I've seen smaller ones do /128. Probably just cluelessness.

The bigger issue is the need for a secure backplane, which will remain until all server software authenticates all sockets in a strong way.

Re: One way to make containers network: BGP

#12

Post by jeff_marshall »

api wrote: post #11
jeff_marshall wrote: post #10

Earlier quoted context omitted.

Which provider won't give you a /64 (that supports IPv6 at all)? Even my residential cable provider gives me a /60. Anything less seems absurd.

I've seen smaller ones do /128. Probably just cluelessness. The bigger issue is the need for a secure backplane, which will remain until all server software authenticates all sockets in a strong way.

I agree re: the secure backplane (though I would prefer to see it done outside server software in a more comprehensive manner).

I'm surprised that the bar isn't set a bit higher inside the cloud provider infrastructure for tenant separation at the network level. I suppose it boils down to the lack of assurance at an even lower level (who trusts Xen these days?) that seems unlikely to be fixed in the short term.

Re: One way to make containers network: BGP

#14

Post by jaytaylor »

crb wrote: post #9

On the topic of "why do we need a distributed KV store for an overlay network?" from the blog: there's a good blog post about why Kubernetes doesn't use Docker's libnetwork. http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-...

Thanks! I had almost forgotten about the container networking mayhem. Would love to find out what progress has been made over the past six months since the blog post was written.

Re: One way to make containers network: BGP

#15

Post by cthalupa »

There's a lot of misinformation in this.

>A Linux container is a process, usually with its own filesystem attached to it so that its dependencies are isolated from your normal operating system. In the Docker universe we sometimes talk like it's a virtual machine, but fundamentally, it's just a process. Like any process, it can listen on a port (like 30000) and do networking.

A container isn't a process. It's an amalgamation of cgroups and namespaces. A container can have many processes. Hell, use systemd-nspawn on a volume that contains a linux distro and your container is basically the entire userspace of a full system.

>But what do I do if I have another computer on the same network? How does that container know that 10.0.1.104 belongs to a container on my computer?

Well, BGP certainly isn't a hard requirement. Depending on how you've setup your network, if these are in the same subnet and can communicate via layer 2, you don't need any sort of routing.

>To me, this seems pretty nice. It means that you can easily interpret the packets coming in and out of your machine (and, because we love tcpdump, we want to be able to understand our network traffic). I think there are other advantages but I'm not sure what they are.

I'm not sure where the idea that calico/BGP are required to look at network traffic for containers on your machine came from. If there's network traffic on your machine, you can basically always capture it with tcpdump.

> I find reading this networking stuff pretty difficult; more difficult than usual. For example, Docker also has a networking product they released recently. The webpage says they're doing "overlay networking". I don't know what that is, but it seems like you need etcd or consul or zookeeper. So the networking thing involves a distributed key-value store? Why do I need to have a distributed key-value store to do networking? There is probably a talk about this that I can watch but I don't understand it yet.

I think not at all understanding one of the major players in container networking is a good indication it might not yet be time to personally write a blog about container networking. Also absent is simple bridging.

Julia generally writes fantastic blogs, and I know she doesn't claim to be an expert on this subject and includes a disclaimer about how this is likely to be more wrong than usual, but I feel like there was a lot of room for additional research to be done to produce a more accurate article. I understand the blog is mostly about what she has recently learned, and often has lots of questions unanswered... But this one has a lot of things that are answered, incorrectly :(

Re: One way to make containers network: BGP

#16

Post by sargun »

More on this here: https://medium.com/@sargun/a-critique-of-network-design-ff85... -- BGP isn't just about containers. It's about signaling. It's a mechanism for machines to influence the flow of traffic in the network.

This isn't container weirdness. This is because networks got stuck in 2008. We still don't have have IPv6 SLAAC. Many of us made the jump to layer 3 clos fabrics, but stopped after that. My belief is because AWS EC2, Google GCE, Azure Compute, and others consider this the gold standard.

IPv6 natively supports autoconfiguring multiple IPs per NIC / machine automagically*. This is usually on by default as part of the privacy extensions, so in conjunction with SLAAC, you can cycle through IPs quickly. It also makes multi-endpoint protocols relevant.

Containers and bad networking because of the lack of IP / container is a well-known problem, it's even touched on in the Borg paper, briefly: One IP address per machine complicates things. In Borg, all tasks on a machine use the single IP address of their host, and thus share the host’s port space. This causes a number of difficulties: Borg must schedule ports as a resource; tasks must pre-declare how many ports they need, and be willing to be told which ones to use when they start; the Borglet must enforce port isolation; and the naming and RPC systems must handle ports as well as IP addresses.

Thanks to the advent of Linux namespaces, VMs, IPv6, and software-defined networking, Kubernetes can take a more user-friendly approach that eliminates these complications: every pod and service gets its own IP address, allowing developers to choose ports rather than requiring their software to adapt to the ones chosen by the infrastructure, and removes the infrastructure complexity of managing ports.

But, I ask, what's wrong with the Docker approach of rewriting ports? Reachability is our primary concern, and I'm unfortunately BGP hasn't become the lingua franca for most networks ("The Cloud"). I actually think ILA (https://tools.ietf.org/html/draft-herbert-nvo3-ila-00#sectio...) / ILNP (RFC6741) are the most interesting approaches here.

Re: One way to make containers network: BGP

#17

Post by ymse »

jlgaddis wrote: post #13

BGP seems a needlessly complex solution to this problem. VXLAN would, IMO, be a much better fit. (--Network engineer who manages BGP for an ISP)

I don't know. With VX?LANs you need a whole lot of other infrastructure to manage e.g. routes, load balancing or tenant-specific ACLs. By coercing BGP into providing isolation, you can slap on routing, load balancing, ACLs "for free", i.e. managed by the same control plane.

If you just need isolation I agree with you. But I actually find the Calico solution rather elegant when looking at the whole package.

(--System administrator who manages VXLAN on a public cloud)

Re: One way to make containers network: BGP

#18

Post by amouat »

cthalupa wrote: post #15

There's a lot of misinformation in this. >A Linux container is a process, usually with its own filesystem attached to it so that its dependencies are isolated from your normal operating system. In the Docker universe we sometimes talk like it's a virtual machine, but fundamentally, it's just a process. Like any process, it can listen on a port (like 30000) and do networking. A container isn't a process. It's an amalg…

I've never read any of her blogs before (that I can recall) and I do agree there are some misunderstandings. But it was clearly written as as a brain dump to help other people going through the same process and it largely achieves that goal. I really don't like the concept "don't write a blog unless you're an expert" - we would lose out on lots of valuable discussions and helpful articles, especially for beginners. For example, we wouldn't have this HN thread with useful commentary if it wasn't for the author.

I think the best approach is to constructively comment and engage to improve the article. The important thing is to do this in a positive manner so that the author feels they have done something of value and started a conversation. It is surprisingly difficult to do this, and I've certainly failed on occasion, but it is definitely worth trying.

Re: One way to make containers network: BGP

#19

Post by amouat »

moondev wrote: post #8
chrissnell wrote: post #6

It's really not that difficult to network containers. We're using flannel [1] on CoreOS. We're using flannel's VXLAN backend to encapsulate container traffic. We're Kubernetes users so every kube pod [2] gets it's own subnet and flannel handles the routing between those subnets, across all CoreOS servers in the cluster. I was skeptical when we first deployed it but we've found it to be dependable and fast. We're runn…

Is flannel used in Kubernetes for networking by default? Or is it something that needs to enabled and configured separately?

Kubernetes has a requirement that containers (more accurately "pods") can connect via a "flat networking space". How this is achieved varies between deployments, flannel, calico and weave are all common approaches. Kelsey Hightower's "Kubernetes the Hard Way" simply configured it at the router level: https://github.com/kelseyhightower/kubernetes-the-hard-way/b...

Re: One way to make containers network: BGP

#20

Post by tptacek »

Especially since there isn't really a policy-routing component to this, isn't BGP pretty _extremely_ complicated for the problem Calico is trying to solve?

Stipulating that you need a routing protocol here (you don't, right? You can do proxy ARP, or some more modern equivalent of proxy ARP.), there's a whole family of routing protocols optimized for this scenario, of which OSPF is the best-known.

Post reply on HN