Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One way to make containers network: BGP

jvns.ca

Post reply on HN
Print view View original

61–70 of 71 posts

Re: One way to make containers network: BGP

#61

Post by otterley »

feisuzhu wrote: post #57
otterley wrote: post #53

The real problem is that cloud providers don't provide out-of-the-box functionality to assign more than one IP to a network interface. If they did this, there wouldn't even be an issue. I've been requesting this feature from the EC2 team at AWS for some time about this, to no avail. You can bind multiple interfaces (ENIs) to an instance (up to 6, I think, depending on the instance size), each with a separate IP addre…

It's not a waste of cycles. You are just pushing your responsibilities to AWS, the cycles and memory for routing your additional IPs are required anyway.

Why do you think that maintaining the network should be my responsibility, as opposed to the provider's?

Re: One way to make containers network: BGP

#62

Post by otterley »

tw04 wrote: post #55
otterley wrote: post #52

Earlier quoted context omitted.

You don't really need IPv6 to do this. IPv4 is sufficient; you just need to assign more than 1 IP address to your physical interfaces. This has been possible since at least the early 1990s. The problem is that many cloud providers (ahem EC2) don't make this trivially easy like they should.

They don't make it trivial because we're out of IPv4 address space. Which would be the reason for doing this with IPv6. The automatic addressing of IPv4 is also nowhere near as simple as v6, not even close.

Both AWS and GCE can allocate private 10.0.0.0/8 subnets for internal networks (e.g. VPCs). There is no address scarcity in such subnets.

Re: One way to make containers network: BGP

#63

Post by detaro »

tptacek wrote: post #54
detaro wrote: post #32

Earlier quoted context omitted.

BGP IMHO is much simpler than OSPF. No different area types, support for communities, no need to keep a link-state database for the entire network in all nodes, ...

People keep talking about link state database overhead, but how significant is this in reality? The graphs we're talking about, even in huge deployments, are small. If you're running etcd or consul, I'm not sure you retain the right to call LSA flooding "complicated". It's simple compared to RAFT!

Size probably doesn't matter that much until you start to fill entire datacenters, OSPF nowadays should work with hundreds of routers as well. Would be interesting to see how failure cases compare, if I remember right one of the arguments for BGP in data-center fabrics was that the updates following them stay more localized. (EDIT: a description how Microsoft uses it for really large networks, slide 11 talks about surges: http://www.janog.gr.jp/meeting/janog33/doc/janog33-bgp-nkpos...)

I find BGP easier to understand, and I don't see what benefit OSPF would have. (Not that I really have non-trivial experience with either, have only used them at home and toy networks)

Re: One way to make containers network: BGP

#64

Post by bboreham »

sargun wrote: post #47
bboreham wrote: post #33

Earlier quoted context omitted.

> what's wrong with the Docker approach of rewriting ports It requires that you rewrite the software trying to talk to that port, to make it aware that you've put the new port number in a special environment variable.

Have you looked at Docker bridge mode? and Mesosphere's VIPs? What do you think of them?

Docker bridge only works between containers on one machine; this is exactly why we wrote Weave Net two years ago, to let you network simply between containers running anywhere.

I hadn't considered using Virtual IPs to reverse out port-mapping. I guess it would work provided you have good connectivity between hosts - it would be a nightmare to try to configure a firewall where the actual ports in use jump around all the time.

Also such schemes require that you know in advance which ports each component listens on, and that there are no loops in the graph. Both of these requirements can be constraining.

Re: One way to make containers network: BGP

#65

Post by e12e »

otterley wrote: post #62
tw04 wrote: post #55

Earlier quoted context omitted.

They don't make it trivial because we're out of IPv4 address space. Which would be the reason for doing this with IPv6. The automatic addressing of IPv4 is also nowhere near as simple as v6, not even close.

Both AWS and GCE can allocate private 10.0.0.0/8 subnets for internal networks (e.g. VPCs). There is no address scarcity in such subnets.

There's a big difference between routable (on the Internet) addresses and non-routable addresses. For one, you can trivially merge two different sets of resources (say from two different organizations or projects) without address conflicts, if they both have globally unique, routable addresses.

LAN networking is great. Internet networking is so much better, it has effectively given birth to a new technological age.

Re: One way to make containers network: BGP

#66

Post by NetStrikeForce »

q3k wrote: post #43
NetStrikeForce wrote: post #39

Or you could NAT on the host and deploy simpler overlay networking: https://github.com/pjperez/docker-wormhole You can deploy this on any machine (container or not) and have it always reachable from other members of the same network, which could be e.g. servers on different providers (AWS, Azure, Digital Ocean, etc)

You should probably mention that this is a PaaS. (and maybe also that you are affiliated with them)

Hi,

Sorry, I should have made it explicit. As it's my own repo and my profile's email address gives away I'm part of Wormhole I didn't think about making a statement on the coment; but you're right.

Thanks!

Re: One way to make containers network: BGP

#67

Post by otterley »

e12e wrote: post #65
otterley wrote: post #62

Earlier quoted context omitted.

Both AWS and GCE can allocate private 10.0.0.0/8 subnets for internal networks (e.g. VPCs). There is no address scarcity in such subnets.

There's a big difference between routable (on the Internet) addresses and non-routable addresses. For one, you can trivially merge two different sets of resources (say from two different organizations or projects) without address conflicts, if they both have globally unique, routable addresses. LAN networking is great. Internet networking is so much better, it has effectively given birth to a new technological age.

I don't disagree on these particular points. My contention is solely that lack of IPv6 is not a showstopper.

Re: One way to make containers network: BGP

#68

Post by X-Istence »

hueving wrote: post #59
X-Istence wrote: post #41

Earlier quoted context omitted.

Calico is installing routes in the Linux kernel. Those routes are pulled out and distributed using BIRD. BIRD can do OSPF instead if you'd like. All Calico cares about is that routes are distributed across various systems, they don't necessarily care how you do it (configure BIRD however you'd like). BGP is surprisingly simple and easy to set up with BIRD. Setting up a route reflector with local hosts on the same L2…

>Why do people think BGP is complicated? Read your own paragraph before this question. Why do I need to run another process to exchange routes and configure a mesh or a route reflector? As an admin that's just another mess of processes and communication to worry about. Just because BGP is easy for you does not mean it's easy for most server admins and devs without heavy networking backgrounds.

Wait what? How else should we be exchanging routes? Should we shove them into a distributed key value store and then having each of the nodes pull out the routes and installing them?

> As an admin that's just another mess of processes and communication to worry about.

Yet we fully expect admins to understand and build HA redundant clusters for databases, or how to manage and update all the machines under their control, and a variety of other tasks.

There is nothing inherently different about running a BGP speaking daemon. It's all config.

I don't have a heavy networking background at all. I'm a software engineer that's currently working as a system architect, but even I can understand something as simple as a route distribution system.

Re: One way to make containers network: BGP

#69

Post by X-Istence »

tptacek wrote: post #50
X-Istence wrote: post #41

Earlier quoted context omitted.

Calico is installing routes in the Linux kernel. Those routes are pulled out and distributed using BIRD. BIRD can do OSPF instead if you'd like. All Calico cares about is that routes are distributed across various systems, they don't necessarily care how you do it (configure BIRD however you'd like). BGP is surprisingly simple and easy to set up with BIRD. Setting up a route reflector with local hosts on the same L2…

Because BGP is complicated compared to intradomain routing protocols.

In this case you are using BGP only for it's ability to send routes from one place to another.

This isn't complicated, it's config management. You can ignore 99% of what BGP can do in this use case.

Re: One way to make containers network: BGP

#70

Post by tw04 »

otterley wrote: post #67
e12e wrote: post #65

Earlier quoted context omitted.

There's a big difference between routable (on the Internet) addresses and non-routable addresses. For one, you can trivially merge two different sets of resources (say from two different organizations or projects) without address conflicts, if they both have globally unique, routable addresses. LAN networking is great. Internet networking is so much better, it has effectively given birth to a new technological age.

I don't disagree on these particular points. My contention is solely that lack of IPv6 is not a showstopper.

It absolutely is a showstopper if you don't have applications that are NAT friendly. And in 2016 if you are forcing people to use NAT and claiming that's a fix, you're doing the entire world a disservice.
Post reply on HN