11–19 of 19 posts
I understand you are only testing for performance but it's worth mentioning that VXLAN does encapsulation and allows for a wider set of use-cases than the solutions that require cluster nodes share a L2. Other than that, great benchmarking! IPvlan is pretty neat, it's the natural evolution of macvlan and hopefully catches on quickly.
I've been a big fan of ipvlan since it was added to the kernel[1] mainly for the reasons you've outlined and the overall reduction in complexity compared to bridges and overlay networks. It should also be noted that ipvlan offers a bit of an improvement over macvlan because ipvlan also works in places where L2 is not an option (i.e cloud providers). In the post you mentioned one of the drawbacks to adopting ipvlan wa…
I'd like to see Joyent's "Fabric" VLANs in this list. They have been very easy to administrate and seem quite performant.
We have been using calico Cni with kubernetes with quite some success. It's a solution that assigns ips and then sets up bgp routing between your docker/kubernetes nodes. It makes it very easy to trace what's happening. Not as easy as a flat lan but still pretty easy. Netstat -rn and you can see where the traffic is going or coming from. The added benefit is you can then bgp peer with other clusters and get routing a…
For small configurations flannel/host-gw is OK, I guess, but if one have several kubernetes clusters, flannel/host-gw becomes harder to maintain. That's a place where calico should be useful.
Hi! I'm one of the authors of this research. Feel free to ask any questions, I'll be happy to tell you more.
I'm not sure if you're focusing on bare metal, but Kubernetes can assign IPs itself, and does so out-of-the-box on AWS, GCE, GKE, Azure & OpenStack (and probably others as well). On AWS it uses the equivalent to flannel's aws-vpc by default (and one of my personal goals is to make it easy to install one of the networking options that goes to bigger scale, so I greatly appreciate your guide here!) So you don't need a…
Our original intention in doing this research is to answer the question "how much do we lose in terms of latency and throughput?" So, as you've said, flannel's aws-vpc is equivalent to out of the box k8s/AWS solution, flannel's host-gw is equivalent to calico. Since they use same "backend", in benchmarks we focused on flannel because it's easier to switch in flannel from one configuration to another (basically just put new config via etcdctl and restart the daemons).
However, thanks in pointing it out. In the article we should state our motivation clearer and add references to another solutions, such as calico.
I understand you are only testing for performance but it's worth mentioning that VXLAN does encapsulation and allows for a wider set of use-cases than the solutions that require cluster nodes share a L2. Other than that, great benchmarking! IPvlan is pretty neat, it's the natural evolution of macvlan and hopefully catches on quickly.
Also, VXLAN offload is in almost all modern NICs.
Hi! I'm one of the authors of this research. Feel free to ask any questions, I'll be happy to tell you more.
We have been using calico Cni with kubernetes with quite some success. It's a solution that assigns ips and then sets up bgp routing between your docker/kubernetes nodes. It makes it very easy to trace what's happening. Not as easy as a flat lan but still pretty easy. Netstat -rn and you can see where the traffic is going or coming from. The added benefit is you can then bgp peer with other clusters and get routing a…
Calico looks good. As far as we understand calico works the same way as flannel/host-gw does, i.e. creates a route for all subnets in the cluster. Of course, calico is more advanced technology than flannel/host-gw, but the reason why we haven't tested it is following: they're different in control path but the same in data path. Our original intention was to test underlying linux-kernel mechanisms and understood how m…
Earlier quoted context omitted.
Also, VXLAN offload is in almost all modern NICs.
Does this mean, that NICs computes checksums inside vxlan payload? As far as I understand, the major additional cost of vxlan comes from additional passing thought the whole network stack, which includes iptables and routing.