Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One way to make containers network: BGP

jvns.ca

Post reply on HN
Print view View original

31–40 of 71 posts

Re: One way to make containers network: BGP

#31

Post by bboreham »

delinka wrote: post #22

Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router. BGP is the process by which ranges of IPs are claimed by routers. Is Calico reall…

People are networking their containers on IP address ranges segregated from their host addresses. So the routers have no idea where to send the packets, unless you take some additional steps.

Re: One way to make containers network: BGP

#32

Post by detaro »

lobster_johnson wrote: post #25

BGP looks really complex. Isn't OSPF (BGP's "little brother") a much attractive choice here? It's still complex, but should be much simpler. Another attractive alternative to Flannel is Weave [1], run in the simpler non-overlay mode. In this mode, it won't start a SDN, but will simply act as a bridge/route maintainer, similar to Flannel. [1] https://www.weave.works/products/weave-net/

BGP IMHO is much simpler than OSPF. No different area types, support for communities, no need to keep a link-state database for the entire network in all nodes, ...

Re: One way to make containers network: BGP

#33

Post by bboreham »

sargun wrote: post #16

More on this here: https://medium.com/@sargun/a-critique-of-network-design-ff85... -- BGP isn't just about containers. It's about signaling. It's a mechanism for machines to influence the flow of traffic in the network. This isn't container weirdness. This is because networks got stuck in 2008. We still don't have have IPv6 SLAAC. Many of us made the jump to layer 3 clos fabrics, but stopped after that. My belief is…

> what's wrong with the Docker approach of rewriting ports

It requires that you rewrite the software trying to talk to that port, to make it aware that you've put the new port number in a special environment variable.

Re: One way to make containers network: BGP

#35

Post by iheartmemcache »

delinka wrote: post #22

Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router. BGP is the process by which ranges of IPs are claimed by routers. Is Calico reall…

Yeah, have an upvote. This is totally a case of "using an industrial core drill with tungsten carbide bits when all you need is an Ikea drill and a chinese cheese-grade bit".

Sounds like this guy just found out about a cool new tool and decided to blog about it. BGP can be used on local networks but it's total overkill for a docker situation where all of your instances are likely in the same rack (often on the same machine!). Ifyou don't even have an AS from ARIN/RIPE there's no reason to even touch this (as you pointed out, it's all the protocol is designed to broadcast to the public internet -- e.g. 'hey I own this AS which has rights to this net-block, direct packets in this fashion please!' Jeez.

I have no idea what the CPU overhead of running this is but I'm sure it's not trivial, especially if the BGP daemon is tuned to retain any significant amount of the whole BGP table (RAM/swap issues galore I'd imagine). Granted the article is titled 'one way'.. which is empirically true, it's a Rube Goldberg way of going about networking.

(n.b., OSPF is what people use for "BGP" within your own intranet, even when you have tens of thousands of boxes. It's called "border gate protocol" for a reason..)

(Sorry, I don't use docker so I can't actually make a constructive comment telling you what the canonical/right way of doing it is, but I can assure you, this is not it.)

Re: One way to make containers network: BGP

#36

Post by atombender »

delinka wrote: post #22

Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router. BGP is the process by which ranges of IPs are claimed by routers. Is Calico reall…

That is indeed the use case being solved here.

Kubernetes enforces a specific rule: Each pod (a group of containers) must be allocated its own cluster-routable IP address. This vastly simplifies Docker setups: In a way, it containerizes the network, just like Docker containerizes processes. It's the only sane way to manage containers, in my opinion.

This system requires something that can hand out IPs and ensure that they're routable on every machine. That something can be done in different ways, range from extremely simple to rather complex. For example, you could have something that acts like a bridge and coordinates with other nodes to find available IPs, and simply maintains the routing table on the nodes themselves in sync with this shared database (Flannel can run in this mode). Or you could use an SDN-defined overlay network (e.g. Weave).

Re: One way to make containers network: BGP

#37

Post by mrmondo »

We're just about to switch to BGP internally using Calico (mentioned in another comment, I believe performance is good now), we run around 300-600 containers currently using our implementation using Consul+Serf. We'll drop a talk on it once we've made the switch if anyone is interested. We're deliberately avoiding flannel because of the tunnelled networking and added complexity that we don't feel we want to introduce.

Re: One way to make containers network: BGP

#38

Post by X-Istence »

lobster_johnson wrote: post #25

BGP looks really complex. Isn't OSPF (BGP's "little brother") a much attractive choice here? It's still complex, but should be much simpler. Another attractive alternative to Flannel is Weave [1], run in the simpler non-overlay mode. In this mode, it won't start a SDN, but will simply act as a bridge/route maintainer, similar to Flannel. [1] https://www.weave.works/products/weave-net/

Calico uses BIRD, and Calico just programs routes into the Linux kernel, BIRD then picks those up.

BIRD supports OSPF, so if you'd like to import/export routes using OSPF you can.

Re: One way to make containers network: BGP

#39

Post by NetStrikeForce »

Or you could NAT on the host and deploy simpler overlay networking: https://github.com/pjperez/docker-wormhole

You can deploy this on any machine (container or not) and have it always reachable from other members of the same network, which could be e.g. servers on different providers (AWS, Azure, Digital Ocean, etc)

Re: One way to make containers network: BGP

#40

Post by X-Istence »

iheartmemcache wrote: post #35
delinka wrote: post #22

Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router. BGP is the process by which ranges of IPs are claimed by routers. Is Calico reall…

Yeah, have an upvote. This is totally a case of "using an industrial core drill with tungsten carbide bits when all you need is an Ikea drill and a chinese cheese-grade bit". Sounds like this guy just found out about a cool new tool and decided to blog about it. BGP can be used on local networks but it's total overkill for a docker situation where all of your instances are likely in the same rack (often on the same m…

> (Sorry, I don't use docker so I can't actually make a constructive comment telling you what the canonical/right way of doing it is, but I can assure you, this is not it.)

And you are categorically wrong. This is a very good way to do it.

Post reply on HN