Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One way to make containers network: BGP

jvns.ca

Post reply on HN
Print view View original

51–60 of 71 posts

Re: One way to make containers network: BGP

#51

Post by e12e »

I've for a long time wondered if anyone has successfully just gone full ipv6 only with a substantial container/vm roll-out. On paper it should have:

1) enough addresses. Just enough. For everything. For everyone. Google-scale enough.

2) Good out-of-the box dynamic assignment of addresses.

And finally, optional integration with ipsec, which I get might in the end be over-engineered, and under-used -- but wouldn't it be nice if you could just trust the network (you'd still have to bootstrap trust somehow, probably running your own x509 CA -- but how nice to be able to flip open any book on networking from the 80s and just replace the ipv4 addressing with ipv6 and just go ahead and use plain rsh and /etc/allow.hosts as your entire infrastructure for actually secure intra-cluster networking -- even across data-centres and what not. [ed: and secure nfsv3! wo-hoo]).

But anyway, have anyone actually done this? Does it work (for a meaningfully large value of work)?

Re: One way to make containers network: BGP

#52

Post by otterley »

e12e wrote: post #51

I've for a long time wondered if anyone has successfully just gone full ipv6 only with a substantial container/vm roll-out. On paper it should have: 1) enough addresses. Just enough. For everything. For everyone. Google-scale enough. 2) Good out-of-the box dynamic assignment of addresses. And finally, optional integration with ipsec, which I get might in the end be over-engineered, and under-used -- but wouldn't it b…

You don't really need IPv6 to do this. IPv4 is sufficient; you just need to assign more than 1 IP address to your physical interfaces. This has been possible since at least the early 1990s.

The problem is that many cloud providers (ahem EC2) don't make this trivially easy like they should.

Re: One way to make containers network: BGP

#53

Post by otterley »

The real problem is that cloud providers don't provide out-of-the-box functionality to assign more than one IP to a network interface. If they did this, there wouldn't even be an issue.

I've been requesting this feature from the EC2 team at AWS for some time about this, to no avail. You can bind multiple interfaces (ENIs) to an instance (up to 6, I think, depending on the instance size), each with a separate IP address, but not multiple IPs to a single interface.

BGP, flannel, vxlan, etc. are IMO a waste of cycles and add needless complexity to what could otherwise be a very simple architecture.

Re: One way to make containers network: BGP

#54

Post by tptacek »

detaro wrote: post #32
lobster_johnson wrote: post #25

BGP looks really complex. Isn't OSPF (BGP's "little brother") a much attractive choice here? It's still complex, but should be much simpler. Another attractive alternative to Flannel is Weave [1], run in the simpler non-overlay mode. In this mode, it won't start a SDN, but will simply act as a bridge/route maintainer, similar to Flannel. [1] https://www.weave.works/products/weave-net/

BGP IMHO is much simpler than OSPF. No different area types, support for communities, no need to keep a link-state database for the entire network in all nodes, ...

People keep talking about link state database overhead, but how significant is this in reality? The graphs we're talking about, even in huge deployments, are small.

If you're running etcd or consul, I'm not sure you retain the right to call LSA flooding "complicated". It's simple compared to RAFT!

Re: One way to make containers network: BGP

#55

Post by tw04 »

otterley wrote: post #52
e12e wrote: post #51

I've for a long time wondered if anyone has successfully just gone full ipv6 only with a substantial container/vm roll-out. On paper it should have: 1) enough addresses. Just enough. For everything. For everyone. Google-scale enough. 2) Good out-of-the box dynamic assignment of addresses. And finally, optional integration with ipsec, which I get might in the end be over-engineered, and under-used -- but wouldn't it b…

You don't really need IPv6 to do this. IPv4 is sufficient; you just need to assign more than 1 IP address to your physical interfaces. This has been possible since at least the early 1990s. The problem is that many cloud providers (ahem EC2) don't make this trivially easy like they should.

They don't make it trivial because we're out of IPv4 address space. Which would be the reason for doing this with IPv6. The automatic addressing of IPv4 is also nowhere near as simple as v6, not even close.

Re: One way to make containers network: BGP

#56

Post by [deleted] »

delinka wrote: post #22

Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router. BGP is the process by which ranges of IPs are claimed by routers. Is Calico reall…

[deleted]

Re: One way to make containers network: BGP

#57

Post by feisuzhu »

otterley wrote: post #53

The real problem is that cloud providers don't provide out-of-the-box functionality to assign more than one IP to a network interface. If they did this, there wouldn't even be an issue. I've been requesting this feature from the EC2 team at AWS for some time about this, to no avail. You can bind multiple interfaces (ENIs) to an instance (up to 6, I think, depending on the instance size), each with a separate IP addre…

It's not a waste of cycles. You are just pushing your responsibilities to AWS, the cycles and memory for routing your additional IPs are required anyway.

Re: One way to make containers network: BGP

#58

Post by jvns »

cthalupa wrote: post #15

There's a lot of misinformation in this. >A Linux container is a process, usually with its own filesystem attached to it so that its dependencies are isolated from your normal operating system. In the Docker universe we sometimes talk like it's a virtual machine, but fundamentally, it's just a process. Like any process, it can listen on a port (like 30000) and do networking. A container isn't a process. It's an amalg…

> I think not at all understanding one of the major players in container networking is a good indication it might not yet be time to personally write a blog about container networking.

Honestly, I struggle with when to publish things a lot -- I practically never write about things I understand well, but I do usually write about things that I understand a little better than this post. Consider it an ongoing experiment :)

I really appreciate factual corrections like "a container isn't a process", and I think comment threads like this are a good place for that. I fixed up a few of the more egregious incorrect things.

Re: One way to make containers network: BGP

#59

Post by hueving »

X-Istence wrote: post #41
tptacek wrote: post #20

Especially since there isn't really a policy-routing component to this, isn't BGP pretty _extremely_ complicated for the problem Calico is trying to solve? Stipulating that you need a routing protocol here (you don't, right? You can do proxy ARP, or some more modern equivalent of proxy ARP.), there's a whole family of routing protocols optimized for this scenario, of which OSPF is the best-known.

Calico is installing routes in the Linux kernel. Those routes are pulled out and distributed using BIRD. BIRD can do OSPF instead if you'd like. All Calico cares about is that routes are distributed across various systems, they don't necessarily care how you do it (configure BIRD however you'd like). BGP is surprisingly simple and easy to set up with BIRD. Setting up a route reflector with local hosts on the same L2…

>Why do people think BGP is complicated?

Read your own paragraph before this question. Why do I need to run another process to exchange routes and configure a mesh or a route reflector? As an admin that's just another mess of processes and communication to worry about.

Just because BGP is easy for you does not mean it's easy for most server admins and devs without heavy networking backgrounds.

Re: One way to make containers network: BGP

#60

Post by e12e »

tw04 wrote: post #55
otterley wrote: post #52

Earlier quoted context omitted.

You don't really need IPv6 to do this. IPv4 is sufficient; you just need to assign more than 1 IP address to your physical interfaces. This has been possible since at least the early 1990s. The problem is that many cloud providers (ahem EC2) don't make this trivially easy like they should.

They don't make it trivial because we're out of IPv4 address space. Which would be the reason for doing this with IPv6. The automatic addressing of IPv4 is also nowhere near as simple as v6, not even close.

Exactly. Last place I worked, we had two whole routable /24s just for servers - it was still not enough to easily give every service routable addresses (especially not as we moved to microservices, and were considering floating ips in addition to per service-instance ips etc).

People suffer from a serious case of Stockholm syndrome wrt. ipv4 addresses and non-routable networks and what-not. There are some (very few) good use-cases for NAT - in most all other cases it just makes everything more complicated - for no real gain (well, you get to avoid buying networking equipment that supports ipv6...). And don't get me started on the "but it helps with security"-crowd... If you need a firewall, get a firewall. Stop conflating it with accidental features of limited address space.

I would have experimented more with ipv6 on my personal server already, if only I could get broadband that actually supported ipv6 (apparently my DSLAM is from the 90s). But now I'm moving, so hopefully I can get that last bit sorted. If nothing else it would appear most 4g networks support ipv6.

For non-personal, non-limited use, one would probably need to set up a fleet of ipv4 proxies/load-balancers -- but I'd be more than happy if I could just move to ipv6 and stop caring about the rest of the luddites ;-)

The main feature draw (on paper) of ipv6 isn't that it enables anything new, it's just that it allows simple stuff to be simple (again). And radical simplicity can be a great feature.

Post reply on HN