Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparison of Networking Solutions for Kubernetes

machinezone.github.io

Post reply on HN
Print view View original

1–10 of 19 posts

Re: Comparison of Networking Solutions for Kubernetes

#3

Post by jpgvm »

I understand you are only testing for performance but it's worth mentioning that VXLAN does encapsulation and allows for a wider set of use-cases than the solutions that require cluster nodes share a L2.

Other than that, great benchmarking! IPvlan is pretty neat, it's the natural evolution of macvlan and hopefully catches on quickly.

Re: Comparison of Networking Solutions for Kubernetes

#4

Post by justinsb »

moigagoo wrote: post #2

Hi! I'm one of the authors of this research. Feel free to ask any questions, I'll be happy to tell you more.

I'm not sure if you're focusing on bare metal, but Kubernetes can assign IPs itself, and does so out-of-the-box on AWS, GCE, GKE, Azure & OpenStack (and probably others as well). On AWS it uses the equivalent to flannel's aws-vpc by default (and one of my personal goals is to make it easy to install one of the networking options that goes to bigger scale, so I greatly appreciate your guide here!)

So you don't need a third party networking solution, but Kubernetes is open so a number have been nicely integrated. flannel is the most popular choice, but there are many others (I hear about Calico a lot)

Re: Comparison of Networking Solutions for Kubernetes

#7

Post by kelseyhightower »

I've been a big fan of ipvlan since it was added to the kernel[1] mainly for the reasons you've outlined and the overall reduction in complexity compared to bridges and overlay networks. It should also be noted that ipvlan offers a bit of an improvement over macvlan because ipvlan also works in places where L2 is not an option (i.e cloud providers).

In the post you mentioned one of the drawbacks to adopting ipvlan was the lack of tooling to manage it. To address this issue, and a few others[2], Kubernetes has recently adopted the Container Network Interface(CNI)[3] standard as the solution for managing network plugins in Kubernetes. CNI ships with a few plugins including ipvlan and, maybe more importantly, the ability to allocate and manage IP addresses[4]. Over time CNI should give Kubernetes the flexibility to work well with multiple container runtimes including Docker and rkt, while allowing third party solutions to fill in the gaps around networking.

[1] https://www.kernel.org/doc/Documentation/networking/ipvlan.t...

[2] http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-...

[3] https://github.com/appc/cni/blob/master/SPEC.md

[4] https://github.com/appc/cni/blob/master/SPEC.md#ip-allocatio...

Re: Comparison of Networking Solutions for Kubernetes

#9

Post by tobad357 »

We have been using calico Cni with kubernetes with quite some success. It's a solution that assigns ips and then sets up bgp routing between your docker/kubernetes nodes. It makes it very easy to trace what's happening. Not as easy as a flat lan but still pretty easy.

Netstat -rn and you can see where the traffic is going or coming from. The added benefit is you can then bgp peer with other clusters and get routing across them

http://www.projectcalico.org

Re: Comparison of Networking Solutions for Kubernetes

#10

Post by chris_marino »

Great post!

The results of these benchmarks do not surprise me at all. To me, they all fall in to the category of 'less (overhead) is more (performance)'. With VXLAN encap being the obvious example of greatest overhead.

I think its also worth mentioning that k8s networking is being enhanced in v1.2 to support isolation and multi-tenancy through ThirdParty resources (back end network solutions). The alternatives included in the benchmarks aren't going to be able to support these kinds of network policy as is.

And, unfortunately, things get a more complicated when you want to provide more than simple reachability (which is all that k8s asks for today). The tradeoff is to be able to control the packets with the lowest overhead possible. VXLANs will give you isolation, but at the cost of encapsulation. Stacking bridges and tunnels and distributing VNIDs/routes not only introduces more latency, but becomes another multi-host coordination problem (matching tunnel IDs, etc).

We're working on a new way to build cloud native networks that avoids the encap, but still lets you control all the packets.

You can learn more at http://romana.io if you're interested.

Post reply on HN