Announcing Intel Clear Containers V2.1.1 • Hacker News Board

Re: Announcing Intel Clear Containers V2.1.1

#11

Post by chatmasta » Tue, Mar 14, 2017, 11:21 AM UTC

doublerebel wrote: post #10

chatmasta wrote: post #3

I hadn't seen this project before. It looks really cool. I especially like the support for pushing network configuration at startup (via the "hyperstart" concept in v1 and systemd in v2). This is sorely lacking in Docker. You can accomplish it with pipework (which is just a wrapper around `ip exec` in the container netns), but then you need to write code in the container like "wait for interface XX to be up before ru…

FWIW I just use Joyent Triton with SmartOS and docker containers, my VLANs exist on start and I can't imagine dealing with those issues. What SDN solution are you using that requires this kind of init process? Solaris solved the service dependency graph problem with SMF, which is nice.

I'm using out of the box docker container networking. I setup a bridge network (docker network create) for every external IP on the host. Each container needs multiple external IPs, so it joins multiple bridge networks and then sets up routing rules for the outbound traffic on each interface. When creating a container, you can only join it to one network. But once it's created you can run 'docker network attach' to join other networks (and get an interface for each). However the application is already running at that point.

What I really want is to create and configure a network namespace using my own tools, and then launch a docker container into the existing namespace. I haven't figured out how to do this yet.

Re: Announcing Intel Clear Containers V2.1.1

#12

Post by dlespiau » Tue, Mar 14, 2017, 12:38 PM UTC

chatmasta wrote: post #11

doublerebel wrote: post #10

Earlier quoted context omitted.

FWIW I just use Joyent Triton with SmartOS and docker containers, my VLANs exist on start and I can't imagine dealing with those issues. What SDN solution are you using that requires this kind of init process? Solaris solved the service dependency graph problem with SMF, which is nice.

I'm using out of the box docker container networking. I setup a bridge network (docker network create) for every external IP on the host. Each container needs multiple external IPs, so it joins multiple bridge networks and then sets up routing rules for the outbound traffic on each interface. When creating a container, you can only join it to one network. But once it's created you can run 'docker network attach' to j…

(Clear Containers dev here)

I believe a "correct" way to do it would be to create a network plugin (or extend an existing one that does 80% of what you want) for docker doing what you want. Those are the ones responsible for setting up the network namespace at startup. A new CNM plugin, only if the current one isn't enough of course, a number of things can be done before container startup, eg. https://docs.docker.com/engine/userguide/networking/default_....

The Clear containers OCI runtime doesn't add a feature to push network configuration in addition to what Docker already does today. It relies on a CNM plugin to setup the network namespace, as usual, then reads back the netns configuration to decide how to best interface between the container netns and the network stack within the VM. It's not exactly easy to get right and we have to juggle with all thid because we're trying to shoehorn a VM where netns are the de-facto interface. Work is going on to make all this simpler and integrated, but it will take some time.

A current limitation is that only the netns configuration at startup time is taken into account and `docker network connect` on a running container doesn't work (we'd have to listen to changes in the netns and propagate the configuration to the VM).

A bit more info there: https://github.com/01org/cc-oci-runtime/blob/master/document...

Re: Announcing Intel Clear Containers V2.1.1

#13

Post by chatmasta » Tue, Mar 14, 2017, 3:11 PM UTC

dlespiau wrote: post #12

chatmasta wrote: post #11

Earlier quoted context omitted.

I'm using out of the box docker container networking. I setup a bridge network (docker network create) for every external IP on the host. Each container needs multiple external IPs, so it joins multiple bridge networks and then sets up routing rules for the outbound traffic on each interface. When creating a container, you can only join it to one network. But once it's created you can run 'docker network attach' to j…

(Clear Containers dev here) I believe a "correct" way to do it would be to create a network plugin (or extend an existing one that does 80% of what you want) for docker doing what you want. Those are the ones responsible for setting up the network namespace at startup. A new CNM plugin, only if the current one isn't enough of course, a number of things can be done before container startup, eg. https://docs.docker.com…

Thank you very much for the reply.

I've already seen those links, but the information in your comment provides valuable context. Thanks!

One problem I've noticed is that you cannot add a routing table to a network namespace with the `ip exec netns ` command, because there is no `ip` command for creating a routing table. You need to edit the /etc/rt_tables file. Because of this, if your network configuration depends on creating routing tables, you need to wait for the filesystem to mount so you can edit /etc/rt_tables.

I've tried doing things like `ip exec netns echo "tblnm 42" >> /etc/rt_tables` but I couldn't get it to work (how to redirect echo output in exec subshell into /etc/rt_tables in network namespace?).

I'm trying to create a really fast, multi-tenant routing fabric. I am relying on namespaces to separate rules, routes, and subnets from each other. This way all routing logic for tenants is separate but still done with native linux routing features at kernel speed.

I would love to be able to create the network namespace without any application running in it (so I can take advantage of kernel routing speed), and only launch an application when necessary (hmm... perhaps a one-time application to configure /etc/rt_tables)